Science plays two — a retraction, and concern issued about genetics papers

It’s a busy week at Science. The journal is retracting a controversial paper about which it had previously expressed doubts, and has published an “Expression of Concern” about another article that looks like it might be headed for the same fate.

First, the retraction.

The move involves an October 2009 paper, on which we’ve previously posted, by European researchers who claimed to have made a major advance in the ability to watch how enzymes behave in cells — thereby giving scientists a new tool for monitoring the function of genes.

But back in December, Science editor Bruce Alberts issued this “Expression of Concern” about the research:

[S]erious questions have been raised about the methodsand data presented in this article. The questions focus in particularon the synthesis of the dye-labeled metabolites that are centralto the microarray technique. In addition, the spectroscopicdata the authors cite in support of their claim were not posted to the Bangor University School of Biological Sciences Web siteat the time of publication, despite the authors’ indicationin the Supporting Online Material that the data would be soposted. In response to inquiries from Science, the authors haveprovided new descriptions of the synthetic methods that differsubstantially from those in their published article. Based onour original concerns and the authors’ response, Science hasrequested evaluation of the original data and records by officialsat the authors’ institutions: These officials have agreed toundertake this task.

When we touched on this matter earlier this fall, Science had asked the researchers to retract their article. Now they have:

To our profound regret, peer inspection of the paper after publication revealed errors and omissions in the information provided on the chemistry underlying array compound synthesis, and the processing of array data obtained. After an investigation, the Ethics Committee of the CSIC in Madrid has recommended the withdrawal of the paper. Given the errors in the paper, and the skepticism about the array that they have generated, we retract the paper. We apologize to Science, our institutions, and the scientific community for any inconvenience caused by our paper and its retraction.

Manuel Ferrer, the senior author of the paper, tells us by e-mail that his group stands by its method.

Well known international companies have successfully used the metabolic array technology and that we continue using it. At the moment we are working in two complementary lines. The first one is to make the array available for research academic laboratories and to provide to international well recognized companies. In both cases, the technology have been successfully employed. In the last case the CSIC approved the testing assays by companies. The second is that some of the authors started a[n] exhaustive analysis of the data and we are doing additional analyses to get published again the chemical part of the paper.

As I mentioned several times, I agree with the fact that some errors were in the paper, although does not means that the technology is invalid. This was the reason to retract the paper. It is unclear still for me the CSIC reasons to ask for retraction.

The study has been cited 15 times, according to Thomson Scientific’s Web of Knowledge.

Now for the “Expression of Concern.” (A a bit about that construction: a frown or cocked eyebrow from your teacher is an expression of concern; a letter from the editor of Science casting doubt on the validity of a manuscript is more like a suspension conference with the principal and your parents).

This story is hardly new. In July, researchers at Boston University reported in Science that they had found certain gene variants, called single-nucleotide polymorphisms, or SNPs, linked to Methuselah-like longevity.

As they explained in the abstract to their paper, “Genetic signatures of exceptional longevity in humans:”

we undertook a genome-wide association study of exceptional longevity (EL) in 1055 centenarians and 1267 controls. Using these data, we built a genetic classification model that is based on 150 single-nucleotide polymorphisms (SNPs).

The Boston group, led by Paola Sebastiani, claimed to have found that 90% of centenarians “can be grouped into 19 clusters characterized by different combinations of SNP genotypes — or genetic signatures — of varying predictive value.” In centenarians, they added, the different SNP signatures appeared to either prevent or to delay the onset of age-related illnesses  such as dementia, high blood pressure and cardiovascular disease.

Not surprisingly, the findings made a splash in the lay media. But a letter this week from Bruce Alberts, editor-in-chief of Science, tries to walk back the initial report. (The study is too new to have shown up as cited in Thomson Scientific.) Other experts in genome-wide associations “have expressed technical concerns” about the paper, Alberts writes.

In their study, Sebastiani et al. used a number of different genotyping platforms and neglected to perform data quality-control steps, which resulted in their reporting several false-positive single-nucleotide polymorphism (SNP) associations.

The major flaw, Alberts continues, lies with the researchers’ use of a device called the Illumina 610-Quad array. What’s wrong with it? According to Alberts, the instrument “has been shown in unpublished studies by other investigators to produce artifactual genotype data at a subset of SNPs.”

Artifacts sound bad — much worse than this story about the 610-Quad when Illumina unveiled the device back in 2008. But as soon as the Science article appeared online, experts began casting doubt on the findings, saying they looked too good to be true (how familiar does that sound?). And less than a week after the Boston group’s article appeared online, Newsweek reporter Mary Carmichael reported that Illumina’s the way the researchers used the DNA chips was suspect.

Carmichael quoted Duke geneticist David Goldstein — who evidently brought her the story of the potential flaw with whom she discussed the study —  as suggesting that false-positives led the Boston group astray. Those false-positives could have resulted from the researches having used two different Illumina DNA chips for their analysis, Goldstein told Carmichael.

In laying out the case against the BU findings, Carmichael’s article took a few swipes at Science, including this one:

The flaw in question could be easily addressed with a little follow-up research. In very simplified terms, all that’s needed is for someone to rerun the analysis using a single different DNA chip. But this should have been done already, before publication. The fact that it wasn’t raises the question of how a paper with a missing piece like this got approved and published by Science.

How, indeed? We’ll ask Alberts that very question. Meanwhile, he states:

Science and the authors are taking these concerns seriously. Since learning of these potential problems, Sebastiani et al. have been performing a thorough quality control analysis on the original raw data, as well as generating new data to compare the genotype calls from the 610-Quad array and the other platforms within the same individuals. These steps aim to eliminate biases between platforms. Furthermore, they are undertaking an additional validation measure on several SNPs via the TaqMan® assay, a non–microarray-based genotyping method.

After ensuring that all data are clean, they will redo the statistical and modeling analyses, which they expect to be completed in December. At that point, Science will re-evaluate the paper, determine the extent to which the strength of its original conclusions has been altered by the revised data, and take the appropriate action.

Better late than never, right?

We should note that both the retraction and the expression were included in the materials sent under embargo to reporters this week — which we applaud. Other journals — we’re looking at you, PNAS and Nature — please take note.

Note: Mary Carmichael emailed us to clarify that David Goldstein did not “bring her the story,” as we’d stated, but that he was the “first of several sources I interviewed who brought up the general problem of potential “batch effects,” or artifacts that can arise from using different chips in different frequencies of cases and controls.”

Also, Carmichael points out that all DNA chips, not just Illumina’s, are prone to false positives. The heart of the matter in the B.U. study, she says, is that the researchers used different Illumina chips when analyzing their cases and controls.

0 thoughts on “Science plays two — a retraction, and concern issued about genetics papers”

  1. The phraseology used by the senior author in his e-mail rings some alarm bells in my mind.
    “Well known international companies have successfully used…” he writes and goes on to use the phrases
    1. “…and to provide to international well recognized companies.”, and
    2.”…some errors were in the paper, although does not means that the technology is invalid.”
    I do not pretend to understand the subject matter of this paper but I find the juxtaposition of “international well recognised companies” with protestations about the “(validity of)technology” a little disturbing.
    Is there a commercial interest here(over and above the usual search for funding)? and are there any commercial consequences of a retraction or a non-retraction?

  2. Hi Adam,

    Thanks for this — fascinating news. I’m tickled to show up in one of my favorite blogs (thankfully not because I’m having something retracted). However, I do want to clarify a couple of details about the longevity paper.

    First, re Goldstein: He was the first of several sources I interviewed who brought up the general problem of potential “batch effects,” or artifacts that can arise from using different chips in different frequencies of cases and controls. But I don’t think it’s quite accurate or fair to him to say he “brought me the story,” which suggests he was actively shopping his objections to the lay media (kind of a no-no for scientists). That’s not how it went down. I called him and others when I saw the longevity results because they were very surprising — you wouldn’t expect to find that many genes with large effects when the sample size is as small as it was — and I wanted some background from complex-trait geneticists who could put those findings in context or even perhaps explain them. You know: If your mother says she loves you and also that there are 77 genes associated with longevity, check it out.

    I also want to emphasize a point that gets almost universally misconstrued — this story is not really about faulty technology. ALL chips occasionally identify a few nucleotides wrongly, not just the 610-Quad. There’s no chip that is 100 percent accurate. The problem here is not the use of the 610-Quad, per se. It’s the fact that the cases and the control group were treated differently, so it’s impossible to say whether that difference in treatment was what caused the results or whether the SNP associations were “real.” The way to figure that out is to go back and do a replication study on the resulting SNPs using a single technology (i.e., treat cases and controls the same). It sounds like Sebastiani et al are doing that now for some of their results, which is great to hear, and they should be applauded for it.

    I will be very interested to hear Science’s response to your query. I called the journal back in July but did not get a response. I agree, however, that it’s very good of Science to send its retractions/expressions to reporters, and that other journals ought to follow that lead.

    Thanks again,


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.