Yesterday, Science published two papers which undercut an earlier paper in the journal claiming to show evidence for an arsenic-based strain of bacteria. Guest poster David Sanders, a structural biologist at Purdue University who was involved in a Retraction Watch story in May, argues that the journal could have avoided publishing the rebuttals—a swift retraction of the original was (and still is) the better move.
Allow me to apologize from the start. This narrative is not a typical Retraction Watch post, because it contains a number of personal elements. However, it would be hard to separate my perspective from my experience.
I will begin by asserting that, despite Rosie Redfield’s many valuable contributions to refuting the Wolfe-Simon paper that have culminated with the publication of data she and other investigators have obtained, there was no need for Science to publish additional articles. The Wolfe-Simon paper never should have been published. The only responsible action on the part of Science would be to retract the original article.
On December 3, 2010, I was listening to Morning Edition on National Public Radio and heard a story about a bacterium that used arsenic instead of phosphorous phosphorus to live on and that the arsenic had replaced some of the phosphorous phosphorus in the organism’s DNA. Having worked on enzymatic phosphoryl transfer for much of my career (in case you are interested, my collaborators and I discovered that the two-component regulatory system response regulators were phosphorylated on an aspartate residue), I knew that the instability of arsenate esters made the claimed results impossible. Biology may teach us about novel chemistry, but it doesn’t violate the laws of chemistry.
When I arrived at work I went to the Science website and read the article, including the supplementary material. The article has been dissected at length by others (there are eight critical technical comments to it), but I would like to share my initial analysis. There was Figure 1, which purported to demonstrate that the bacteria grew (poorly) in the presence of 40 (!) mM arsenate in the absence of phosphate.
I immediately recognized that the “+As/-P” growth curve resembled that of a bacterium growing with a limiting nutrient (obviously not arsenate—there was many-fold more arsenate present than any nutrient that a microbiologist would include in a growth experiment). There was the statement in the abstract that there were substantial amounts of arsenate in protein. As a graduate student I was in the laboratory of Daniel E. Koshland Jr., where it had been first determined that protein phosphorylation occurred in bacteria, and I can assure you that it is not at very high levels. (Koshland was also editor of Science from 1985 to 1995.) The analytic techniques used to demonstrate incorporation into macromolecules such as DNA were either shoddy or didn’t provide real support to the conclusions.
The most important component of the original article is Supplementary Table S1. There is ICP-MS analysis of the -P/-As and -P/+As media for phosphorous phosphorus and arsenic content. For each of the media there are two “batches.” Nowhere in the original article is there any mention of which batch was used in which experiments (more on this matter later). The critical data are the phosphorus concentrations. In the April 5, 2010, “batch” of -P/-As medium there is 3.7 +/- 0.4 microM phosphorus (note the standard deviations—we will return to them later as well). In the June 11, 2010, “batch” of -P/-As medium there is <0.3 microM phosphorous phosphorus-normally this result would indicate that the amount of phosphorous phosphorus was below the limit of detection of the instrument. In the April 5, 2010, “batch” of -P/+As medium there is 2.7 +/- 0.3 microM phosphorus, whereas in the July 29, 2010, “batch” of -P/+As medium there is 2.9 +/- 0.3 microM phosphorous phosphorus.
These data are summarized in the main text of the article as follows:
The background PO43- in the medium was 3.1 (+- 0.3) microM on average, with or without added AsO43, coming from trace impurities in the major salts (Table S1).
There can be no statistical or scientific justification for this misrepresentation of the data. One cannot average undetectable and 3.7 microM and get “3.1 (+- 0.3) microM on average, with or without added AsO43.”
No explanation is provided about the divergent analyses. If different “batches” of media have different amounts of phosphorous phosphorus for no apparent reasons, then none of the data can be trusted.
The obvious explanation, one that others have suggested subsequently, was that the arsenate was contaminated with phosphate! This fact would explain the “optimal level of growth” of 40 mM. The bacteria were growing on the phosphate contamination!
What is the evidence for this proposition? First, the whole basis of the Mono Lake bacterial experiments is that arsenate chemically resembles phosphate. It does. It is therefore a likely contaminant. In my previous research experience I have run into the facts that commercial sources of S adenosyl methionine are contaminated with S adenosyl homocysteine, that D-amino acids are contaminated with L-amino acids, that radiochemicals may be “radiochemicallly” pure but they are not analytically pure chemicals, and that detergents were commonly contaminated with peroxides.
The Wolfe-Simon article, being an article in Science, did not indicate the source of the arsenate. To fulfill our curiosity at Purdue University we conducted some ICP-MS experiments on a commercial arsenate source and found that it was indeed and unsurprisingly contaminated with phosphate. Not a publishable result. For the Wolfe-Simon paper—bad data, misrepresented data, inconclusive data, contamination. Game over. No need for additional research or articles. Article was self evidently wrong and should never have been published. It should now be retracted.
The article itself is, however, only a small part of the story. As we’ll see in future essays, the case provides an illustration of the abysmal failure of scientific peer reviewers, scientific journals, government and academic institutions, the media and numerous individuals to do their jobs with competence and integrity.
What are the standards and community practices for citing retracted papers? I was under the impression that one of the goals of retraction was to at least discourage any future citations. Because there is now a large amount of discussion one the original paper, it should remain available and citable for people to understand the critiques.
Good question. There’s nothing wrong with citing a retracted paper, and that’s why good retraction practice is to leave it in the record but mark it “retracted.” The problem is when retracted papers are cited as if they were never retracted, a phenomenon John Budd et al describe here: http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/national/2011/papers/retracted_publicatio.pdf
Wasn’t the retraction made to prevent an error from spreading? I don’t think science’s purpose is to punishing errors made or hide the publication. Did anybody discouraged the author of phlogiston (1667) to publish. We are all adult, we know how to distinguish a hoax, from a mistake or a bad writting case.
I believe any mistake in the writting is due to the pressure to publish or obtain a grant. Good case to show to our students. In addition, what ever happend to the discoverers of the “martian rock” that fell in the antarctica?, Nasa got the money to send its robots to Mars. Is this another case of “science support of life in nearby planet”? I think for all these reasons the paper should remain labeled as “retracted” but carefully analyzed with our students.
This is the same mistake that RW made in the earlier article about the Pauling paper. Retraction is not the appropriate response to honest error or ideas that are proposed that turn out to be wrong. Clearly Sanders feels strongly about this paper – and this isn’t my field and so perhaps it is justified – but there are many papers in my field that get published (sometimes in Nature and Science) that are ‘self-evidently’ wrong (at least according to me). Most of the time I’m right – but I am not going to swear blindly that my instincts are infallible (and I’m sure that Sanders will agree that neither are his).
Many, many papers get an initial peer review that doesn’t go as far as the subsequent ‘post peer-review’ (and I’d venture to suggest that no paper of any importance does!). To assume, as Sander’s appears to be doing, that publication in Science must be equated with absolute truth is a fallacy. It is far better that we (as scientists, and the media) get used to the fact that journal peer review is just the first step towards establishing ‘facts’ – and it has always been thus. Furthermore, having a wider audience for why things went wrong is often a key element in the whole field learning from the mistakes and gaining an appreciation for details that they would not otherwise have thought of. I guarantee that more people now know about the problems with arsenate substitutions in DNA than ever did before the initial paper was published.
I fail to see why NASA’s embarrassment over the press release, Wolfe-Smith’s discomfort at seeing an initial idea fail to stand up, or Rosie Redfield’s decision to take time to look into this more deeply, justifies the overboard reaction we are seeing here. This is not the end of the world folks, and no-one has died (except a few bacterial colonies). It’s just one more paper that didn’t stand up – and getting on a high horse of moral indignation because someone else might have been a tougher reviewer is an over-reaction. I doubt very much if Sanders actually wants to be the sole reviewer on all papers in his field, and given that disinclination, it inevitably follows that other people will review according to their own standards – which Sanders (and I) will not always agree on.
PS. Pet peeve, but I hate it when headlines use the generic term ‘scientist’ or ‘scientists’ when they are specifically referencing a particular person. It conveys the impression of an implicit argument from authority.
Thanks for the thoughtful comment and feedback. This is David Sanders’ opinion, of course.
You write that “Retraction is not the appropriate response to honest error or ideas that are proposed that turn out to be wrong.” Just wanted to note, since it keeps coming up, that the Committee on Publication Ethics says that honest error is a perfectly good reason for retraction. From COPE’s retraction guidelines, available at http://publicationethics.org/files/retraction%20guidelines.pdf:
“Journal editors should consider retracting a publication if:
• they have clear evidence that the findings are unreliable, either as a result of misconduct (e.g. data fabrication) or honest error (e.g. miscalculation or experimental error)”
I’m not sure there is actually an argument here.
There are clearly a range of views on the meaning of retraction, ranging from:
1) Paper stays in place, marked Retracted, with a clear and forthright description of the reasons. which can range from honest error to fraud.
to
2) Paper silently disappears. (In one case, a talk not only disappeared, but was edited out of the seminar history as though it had never occurred 🙂 I don’t know if that’s ever happened to a paper.)
So, might David say some more words about what *he* means by retraction?
Fair point about ‘honest error’, but what I was intending to convey was more like honest claims that end up being erroneous. Which is a little different from finding that your computer program for calculating protein folds had a bug and you no longer thought your interpretation was valid.
I think what Sanders is suggesting is that the P contamination issue is indeed equivalent to a bug for a computational paper. Whether or not the claims were honest has nothing to do with it.
I broadly agree with Joe’s comments, and don’t think papers should be retracted for being wrong, unless they’re shown to be fraudulently wrong, or the authors lose confidence in their data/analyses and choose to retract… however I don’t believe this paper should have been submitted in the first place.
Where this publication scenario has gone wrong is in the most important element of peer review, aka “self peer review”. It’s difficult to comprehend that the authors would allow themselves to be seduced by the sexy idea of a completely revolutionary biochemical structure (arsenic substituting for phosphate in phosphodiester bonds of nucleic acids) without properly assessing whether the far more likely scenarios (the bacteria were ekeing out a living on low level phosphate contaminants; the As “associated” with poorly fractionated DNA wasn’t actually covalently bound in the nucleic acid, etc.) were actually the explanation for their observations.
That’s what I find a little depressing. It wouldn’t have been that difficult to do this work properly…and then there would have been no paper.
On the other hand, this episode hasn’t been without value. A lot of people have learned a little bit more about DNA than they otherwise knew, and now virtually everyone knows that arsenic esters are far more labile than phosphate esters in water. The episode is a microcosm of one of the (messy) ways that science works (ideas tested and the “right” answer established; the role of post-publication peer review etc.).
Paul Feyerabend wrote a philosophy of science book called “Against Method” in which he argued that a signifcant part of scientific advance occurs via non-standard approaches and that some of these may border on the anarchic. Whether that’s a useful description, it’s worth remembering that however we arrive at scientific truths these do exist independently of our approaches!
So either some bacteria can utilise arsenic as a phosphate surrogate at limiting phosphate concentrations or they cannot. We expect that it’s unlikely that they are able to do so, and that expectation has been reinforced by this episode. So we have learned something….
I’m with David Sanders here. It’s a built in principle of science that you question your own findings before letting them out onto a global stage. We all know that when you get an amazing result that challenges fundamental principles of biology, 99 times out of 100 it will be an artefact due to some factor you failed to take into account. Failure to consider this is simply poor science. If Nature, Science and the like become filled with such ‘amazing’ findings which turn out simply to be artefacts, followed by subsequent papers which show this to be the case, science as a discipline will essentially become a farce.
Sensible peer-review should be able to pick up such issues; but if it doesn’t for some reason, and the flaw becomes visible post-publication, such papers should be retracted. Retraction is appropriate for unreliable findings, and in this case the findings are based on false assumptions. The authors should have been given the opportunity to retract this article, with the journal doing so if the authors were unwilling.
Yes. Given what some of the authors have said, voluntary retraction seems unlikely right now, but maybe they’ll feel differently in six months. A year. Maybe longer. People come to conclusions in different ways at different times.
Unilateral retraction by the editor is drastic. It should be one of the last resorts. Sanders seems to be arguing it should be one of the first.
Science magazine has a strong emphasis on newness and sexiness, which in my view very often results in crappy work making it in there. Because of the high impact factor, and strong bias against publishing negative results, a single article in Science can lead an entire field astray.
I can’t understand the view of those who advocate a scientific process with no memory. This is what you get when you retract every paper in which the wrong conclusions were drawn from imperfect methods, and every paper in which the incorrect or incomplete conclusions were drawn from the best evidence at the time. The last published paper then has undue weight, and we end up basing important decisions on a few recent findings instead of decades of research.
Thanks for the comment. Can you clarify who is advocating “a scientific process with no memory?” Retracted studies should not disappear, according to COPE, so they would clearly be available to read and inform work:
“Retracted articles should not be removed from printed copies of the journal (e.g. in libraries) nor from electronic archives but their retracted status should be indicated as clearly as possible.”
From http://publicationethics.org/files/retraction%20guidelines.pdf
What happens to the reviewers of papers that are llater retracted? Are they black-listed? Are their names published?
Does it really matter either way if a paper, which “everyone knows” was wrong, is retracted? What difference does it make?
The argument that seems to come up most often in response to that is ‘to leave them unretracted would be to mislead students and members of the public’ but since when has that been a factor in deciding which papers get published? And if it’s not an issue in publication, why retraction?
It concerns the integrity of the journal and of the entire peer review process (IMO). And this was not a simple error in interpretation of data. It was poor implementation of the scientific method and even poorer peer review. If this paper is not retracted, then the entire peer review process becomes meaningless (which some argue, is already the case). If this paper is not retracted, then we should all submit transcripts with questionable data and outrageous interpretations, so we can make headlines and have our 15 minutes of fame.
“If this paper is not retracted, then the entire peer review process becomes meaningless (which some argue, is already the case).”
Over the long run, the scientific process converges on truth. THE LONG RUN is the important part. If you expect every publication to be perfect and infallible, of course you’re going to have a negative view of peer-review and science in general.
“If this paper is not retracted, then the entire peer review process becomes meaningless (which some argue, is already the case)”
But retracting a paper and saying “we should never have published in the first place” are two different things. You may have a case that the latter should happen but that’s not what will happen if it’s retracted. Journals rarely if ever say the latter, even when papers are retracted, they say “On the basis of information that came to light after review….” They dodge that question.
Retracting this paper (but not declaring the original publication a mistake) could actually harm peer review by creating the attitude that peer reviewers can afford to be lenient because if it’s really bad it’ll end up retracted.
Because this paper should not have been accepted for publication, the best way to somehow remedy the situation is to retract it. Moreover, because NASA chose to publicize this piece of research via a press conference, the retraction should also be highlighted in a similar fashion. If not, the readers of the Glamour magazine will still believe that GFAJ-1’s DNA contains arsenic in lieu of phosphorous.
I don’t know if someone brought this up already, but in this article: http://www.washingtonpost.com/national/health-science/journal-retreats-from-controversial-arsenic-paper/2012/07/08/gJQAFQb7WW_story.html
there are some very disturbing statements such:
1. He said (Taner) the authors of the two new papers “may well regret some of their statements” in the future.
2. Wolfe-Simon and her numerous collaborators had made samples of GFAJ-1 broadly available after her initial results caused a storm of controversy, but she and Tainer said they may have been contaminated or modified in transit.
Does the second statement mean that Wolfe-Simon sent out contaminated samples to the people that requested GFAJ-1? How bad can this story be?
Look, this is NASA, they have a licence to cheat and commit scientific fraud. I think it is clear that there were big names lurking in the background – Paul Davies for instance.
Although your quote looks a bit ominous in isolation, taken in context I don’t think it is really disturbing. After all, Taner goes on to say “There are many reasons not to find things — I don’t find my keys some mornings”
Which quite frankly suprises me not at all.
Taner is sort of right. I, for one, did not find my unicorn this morning. Nor the previous one. But it does not mean the unicorn does not exist.
Wolfe-Simon’s defense strategy will be to modify her arsenic claim in order to render it unfalsifiable. After all, it is not possible to prove that there is no DNA molecule in GFAJ-1 containing at least one P-to-As replacement. She will keep on moving the goalposts and require an ever increasing standard of proof from others. The fact that her own standards on which her original claim was built were low will be immaterial.
Perhaps she could be asked to clone one of the enzymes, preferably one without cofactors, involved in the synthesis of ddNTPs, sorry ddNTAs, and then see if she can produce any product in conditions with no phosphate whatsoever.
As a special concession she can just use ordinary NTPs during the cloning stages.
oops, thats dNTPs, not ddNTPs (used in sequencing).
I blame the coffees.
I wouldn’t necessarily blame the reviewers yet – I haven’t seen the reviews. From personal experience I know that one can rip a manuscript to shreds, and still somehow the editor makes a decision to publish the crap. I firmly believe that having reviews accessible with any published paper, as is being done by EMBO among some other journals, would be helpful to readers, who may miss something that the supposed experts, aka reviewers, spotted in the manuscript. It might also help to preserve integrity into the reviewing/editorial process, making public and obvious the cases of rigorous reviewing, as well as the cases where reviewing was shoddy or the editor decided to flag the manuscript through anyway.
It is clear that either the reviewers and/or the editor messed up. I usually get thoughtful and helpful reviews, and have no desire to junk the present process. How can the process be improved. Publishing all reviews would be helpful. I would like to have reviews of a manuscript shared among all reviewers. Both of these efforts would help reviewers to improve!
Yes I agree with having reviews of a manuscript shared amongst reviewers. There are a few (but very few) journals that I know that do this (Biophysical Journal is one; i.e. you’re sent the other reviews after an editorial decision has been made), and it is very helpful and interesting to read these, and it would certainly help to improve the peer-review process if more widely used.
Apart from anything else it indicates that the editors are giving you a little more respect, and involving you more fully in the process, than just “inviting” you to review (and sending you emails to hurry up!), and then leaving you in the dark until you might happen to chance upon the paper in print at some later date….
In Science and other top-bill journals the review process takes two rounds, sometimes three ;-), so the reviewers see all the reviews at round two. This definitely helps, if all of them are helpful and thoughful. But what sometimes happens is when the editor is bent on pushing the sensationalist crap through, despite negative review(s), the author of negative review doesn’t get the editorial decision, the correspondence just dies out and after a while with indignation the ex-reviewer sees the manuscript not recommended for publication appearing in press with the same glaring errors he/she pointed out… Somebody else, more sympathetic, takes his/her place, writes 2 paragraph-long panegyric, which replaces the negative review at the general editorial meeting…
I concur with Prof Sanders that the editors and the referees failed badly. Sad to say, Wolfe-Simon sounds like a person of questionable judgment, if not integrity. As a postdoc her folly might be excused somewhat on the grounds of youth and inexperience, but the senior scientists who co-authored the paper and her postdoctoral mentor have no such excuse.
There is an additional dimension to this episode that Prof. Sanders has addressed only tangentially, namely, that the “gate-keeping” mechanism in journals like Science is brutal. Technically sound, well-communicated and even groundbreaking work is frequently rejected on grounds of insufficient impact – often without peer review. Researchers learn to lick their wounds and move on, accepting that they did not meet the very high standards of impact and generality to merit publication.
However, when these standards are not uniformly enforced – when work that is grossly, obviously flawed like GFAJ-1 makes it through in such outlets because of well-connected authors or incompetent referees or the like – it is natural for scientists to feel aggrieved and cheated.
Leave the incompetent referees alone for a sec – have you ever wondered who these gate-keepers (aka Editors) are? Let’s say one gets a rejection without review from one of them high-profile journals and looks up rejecting Editor in Pubmed – 9 times out of 10 the Editor has 2-3 manuscripts in medium-tier journals from a high-profile, high-power lab as a former graduate student, and then was shoed-in into a top journal by the ex-boss, without a fail Academy member, HHMI, and things like that. Editorial job as a consolation prize for a nice graduate student with no scientific acumen… And it really doesn’t matter that the submitting author has more papers than all the scientific Editors together, including those in the top-bill journals, where the Editors themselves never published. No, once they get the gate-keeping job, with implications for funding, tenure, etc. for the lucky/accepted authors, they don’t have to listen to arguments anymore.
This hardly the case with Science, its a powerful and prestigious job.
Again, it is naive to think anyone was incompetent here. Rather there was collusion to avoid asking the authors to do more definitive experiments.
“Editorial job as a consolation prize for a nice graduate student with no scientific acumen”
Lacking scientific acumen is, I presume, code for too naive or honest to falsify results to advance their career. Although, again in my experience, that is not far from the truth; people who would never falsify find it very difficult to believe people are cheating, whereas people who do falsify results tend to jump to that conclusion very quickly.
Poachers make the best gamekeepers.
Generally it is not the job of the editor to detect fraud and it certainly is not their job to investigate it, that is the responsibility of their institution.
I personally don’t have a lot of sympathy with complaints about not getting into higher profile journals. If your field is immunology and you can get any significant findings published in the Journal of Immunology or similar journals that are indexed by Pubmed and read by everyone in your field, what do you have to complain of?
To littlegreyrabbit
You are not paying attention. No, lacking scientific acumen simply means that after a bouncing as a grad student in a high profile lab and failing to convert this career-launching opportunity (as evidenced by a few non-consequential publications) a newly minted PhD decides to leave the academic path, and instead goes on to become an Editor. Anyway, I don’t think you are getting what I am trying to say, and I am too old to have another round of straw-man arguments.
A grad student can only publish the results implicit in the project given them. If a grad student has only a few “non-consequential publications” that might simply be a reflection that their project only had the potential for some non-consequential findings.
And the grad student successfully and professionally uncovered them.
I am sorry that you can’t get into all the high impact journals you would like and it may well be the fault of dull, failed former grad student turned editor unable to recognise your superior output.
But weren’t the original of journals simply a means to circulate the findings of researchers to those of similar interests? Reflected in the names of the older ones: “Philosophical Transactions of the Royal Society etc “. So provided there is sufficient space in the journal sphere that all good work gets published – and I think there is and more – then thanks to the proficiency of the database indexing functions, all significant new findings will reach their intended audience.
That doesn’t seem a huge problem to me Pymoladdict. One needs to recognize that Nature and Science are science magazines which publish editorial, news and commentary, as well as scientific papers, and which aim to reflect the progression of science across the cutting edges a full range of scientific fields.
So the fact that editorial decisions in Nature, for example, are made by a group of professional editors that have relevant scientific research experience, but have chosen to work in scientific publishing seems perfectly fine.That’s the kind of science magazines that they are. I expect that they do occasionally aim to be provocative in their selection of papers (e.g. the infamous Beneviste paper on “homeopathy” in Nature some years ago).
Since Nature and Science do publish loads of groundbreaking research (as a pymol addict, I’m sure you’re familiar with the astonishing structures of voltage-gated potassium channels, the first Na channel, the structure of the ribosome, the structure of the Na,K ATPase etc. etc. etc. from recent years), it doesn’t seem that problematic that they occasionally go out on a limb.
After all, as a result of the Beneviste homeopathy paper, robust scientific critiques were published, and we’re probably all the more sure that there is no scientific evidence that water retains a “memory” of solutes. Likewise with the arsenic-in-DNA paper; I doubt anyone familiar with this story would say that they haven’t learned a little about DNA, and the relationship between phosphodiester bonds and their arsenic surrogates.
In fact the real losers in this story are likely to be the authors of the study whose reputations are going to take a hit to the extent that their claims are shown to be unsupportable, and that the submission of their paper was made without the self-critique that should be second nature to anyone that wishes to found out the “truth” as opposed to making a splash…
And nobody is the slightest bit concerned that Professor Sanders doesn’t appear to be able to spell ‘phosphorus’?
An interesting post-mortem of the Wolfe-Simon paper:
http://www.usatoday.com/story/tech/columnist/vergano/2013/02/01/arseniclife-peer-reviews-nasa/1883327/ with the authors’ response to reviewers’ comments at
http://www.documentcloud.org/documents/564124-foia2012-nasa-01-dvergano.html
Just who were the reviewers that blindly accepted the main premise of the paper?
I have not read the linked article yet. But let me guess… Wolfe-Simon still stands firmly behind the findings of her study. So does NASA.
You can find the continuation of the argument for retraction at http://www.periodicplayground.com/blog/bp/2013/02/guest-post-david-sanders-why-its-high-time-to-retract-arseniclife