“Neither snow nor rain nor heat nor gloom of night” keeps U.S. mail carriers from delivering your letters and packages, and neither seven retractions, nor being forced to resign, nor malpractice settlements (and an ongoing case) keeps former Duke oncologist Anil Potti from publishing papers.
Potti’s latest effort, “A Pathway-Based Approach to Identify Molecular Biomarkers in Cancer,” appeared last month in the Annals of Surgical Oncology. The subject of the review is the same as many of his now-retracted papers, none of which are cited by the new paper.
The paper took a while to get through peer review and editing. It was first submitted in May 2010, two months before The Cancer Letter reported that Potti had lied on a grant application about being a Rhodes Scholar and his research program really started to unravel.
We asked Potti’s co-author, Marvaretta Stevenson, about the delay, and whether she had any trepidation about publishing a paper with her former Duke colleague. She had published several papers with Potti before things fell apart, none of which have been retracted, and presumably, she could have withdrawn this paper at some point. We asked similar questions of the editor of the journal, and will update with anything we hear back from either of them.
In case you were thinking that the Annals of Surgical Oncology was some backwater journal where pretty much anyone can publish, here’s a “Special Announcement” on the journal’s homepage:
ASO IMPACT FACTOR RISES
The 2010 Impact Factor for Annals of Surgical Oncology has risen to 4.182, the third consecutive annual increase in the journal’s impact ranking. The journal is now ranked 8 of 187 journals publishing in Thomson Reuters’ (formerly ISI) subject category “Surgery,” making it the top ranked oncology journal in surgery
In short, it ain’t no slouch.
By the way, Potti now has his own Wikipedia entry, with which we had nothing to do. We’re probably a bit happier with ours — with which we also had nothing to do — than he is with his.
So, some really basic questions, since I don’t know what happens behind the scenes:
Do you think this journal’s editors are fully aware of Potti’s history?
What are the chances that every detail of this submission – every bit of data and assertion of the work done – was independently validated?
Good questions, Dave.
It’s hard to imagine anyone in the academic oncology community is unaware of Potti’s history. The fact that this paper took so long to be published after the revelations about him began suggests that the editor did know, and applied more than the usual amount of scrutiny. But we don’t know, because we haven’t heard back from him. We’ll update if we do. Keep in mind that this was a review paper, which means there weren’t any actual new findings.
The paper is open-access. I have read the paper. I am a statistician in a medical context. The paper is entirely theoretical, untroubled by data, actual examples, or anything of an actual demonstration of the “ideas” in the paper.
The key comment is “Fundamental to the development of useful expression signatures is the experimental condition for the discovery cohort (or training set). In many instances, these signatures are generated in vitro, using investigative settings that can be carefully controlled to define a specific biological process (Fig. 1) seen in the training set. These gene expression data are then used to identify a pattern of gene expression— a signature—that reflects and, importantly, predicts the phenotype. The extent to which the profile actually reflects the biological state, rather than what would be observed by chance, can be accessed through statistical measures of predictive ability, typically a leave-one-out cross validation procedure.7,8” (Stevenson and Potti, 2011, p.2) Reference 7 is to another Potti paper. Reference 8 is to a paper from the J. Nevins lab. J. Nevins is a Potti colleague who has published with him on numerous occasions.
Later on the same page the authors state “The ability of the signature to accurately predict the phenotype of the set of test samples that is used for validation provides an important measure of the signature’s robustness.” Sort of a “duh – no kidding” comment. This is devoid of operational specificity – exactly how are they going to do this?
Essentially this is a fluff piece. It has little substance, and is mostly a speculative venture. The key information is contained within Baggerly, K. A. & Coombes, K. R. Deriving Chemosensitivity From Cell Lines: Forensic Bioinformatics And Reproducible Research In High-throughput Biology Annals Of Applied Statistics, 2009, 3, 1309-1334), in which the initial Potti methods are thoroughly and completely debunked, due to numerous errors of manipulation, data handling, and general confusion (duplications within the training set). They state “In the case of these results. In the case of these results, we don’t think the approach works. We think stronger evidence (including worked examples of how it works) need to be provided before this approach is used to guide patient allocation in clinical trials. In the case of the clinical trial noted above, assuming the general approach works means that sensitive/resistant label reversal for one of the drugs (pemetrexed) may actually put patients at risk by giving guidance at odds with the truth, whereas assuming the general approach doesn’t work means that little will be learned from the trial or even that it may provide misleading support for the approach given the inclusion of genes that shouldn’t be there (ERCC1, ERCC4). A broader question is whether this approach could work if applied correctly. We don’t think so. We have tried making predictions from the NCI60 cell lines when we step through the process without the errors noted above, and we get results no better than chance.We communicated with the authors extensively in the early phases of our investigation (which we recommend), and shared the fact that we didn’t think it worked before submitting our … ”
Thus, I would not give much credence to this piece.
The first author is an oncology fellow at Duke. I expect participation in some sort of research (even theoretical research like this) and a publication is a requirement of the program. Of course, one wonders why the first author would want to be associated with Dr Potti in the first place.
“ASO IMPACT FACTOR RISES”
From what I understand, it is rather easy to cheat on impact factors. All a journal has to do is generate journal issues in which they publish summaries (= cite) of all the papers they published during the preceding year or so. That will boost impact factors artificially.
I can see that, given his track record, people are highly sceptical concerning new papers of Potti. Still, this is basically an ad hominem discussion. Papers should be judged by their content and not by the history of their authors.
I noticed that too and perhaps the owners will have to do something about it (not sure, forcing registration, for example). The Comment section of the site is becoming more and more ad hominem. Perhaps it tends to attract disgruntled individuals who see the Comment section on this useful site as a forum where they can anonymously throw mud on anyone who they perceive may have achieved more than they have.
I don’t think that Paul Thompson’s comments are ad hominem at all. I admit that we’ve made a lot of “cute” comments lately; perhaps our levity is misguided. However, I assiduously try to avoid saying “he’s a jerk” or the like. In the case of Anil Potti, it is just too tempting to make comments about his personality because of the delusional nature of his web statements and the imaginary Rhodes scholarship, for example. But I don’t think it’s ad hominem to point out that even after the retractions of seven of his papers, Dr Potti is still appearing as an author: in a paper that espouses an approach that doesn’t work. Is it ad hominem to say that he should get a clue and stick to his clinical work?
Mea culpa: I attacked oncologists in general for being overpaid and uncaring. That got me in a lot of trouble. Nonetheless, there are many who hold the opinion that Anil Potti is not an aberration in the field of oncology, and that there is too much money coming from the pharmaceutical industry that is corrupting research.
Conrad, anything based on facts is legit… I tend to agree with you in the case of Potti. But probably he’ll stop only when he is forced to stop.
I think it is worth a reminder that all the money in the world cannot corrupt research unless there is a corrupt researcher involved. Does DoD or NIH money somehow somehow spend differently than Pharma dollars? What about funding from non-profit patient/disease advocacy groups, is it more noble to cook the books when the sponsor is motivated by personal passions instead of commercial incentives? Many Universities spend more on research than second tier Pharma companies and even larger biotechs. Academic biomedical research is Big Business, too.
In my experience, individual Pharma researchers are no more or less ethical than the academics who trained and mentored me. Given the immediate personal and financial consequences of research fraud in Pharma, an argument can easily be made that individual scientists are more likely to commit fraud in Academia than in Big Pharma. Our firings are immediate and are often followed by swift civil action as well. In contrast, it took years to dump tenured fraudsters from my graduate school faculty, and they all were able to secure scientific jobs with benefits within a few years. I had intimate knowledge of one such offense, and its ultimate outcome remains staggering to me, given the soft-pedaling in the ORI report of the incident.
This is an incredibly important time for industry-academic relations. Pharma is poised to spend unprecedented dollars sponsoring Academic research in search of new targets and lead compounds for in-licensing and late-phase testing. Given the enormous costs associated with clinical development of new drugs, Universities had better get their houses in order before signing on the bottom line. Commercial fraud is likely to be subject to a different level of criminal prosecution and civil litigation.
CL
P.S., I am a basic researcher in Pharma, and am posting anonymously due to contractual obligations. I wish this were not necessary.
P.P.S. In doing some background reading on this question, I encountered this question from Karen Wooley on Retraction Watch. As far as I can tell, it remains unanswered:
http://www.retractionwatch.com/2011/09/07/anil-potti-failed-to-disclose-corporate-ties-in-yet-to-be-retracted-jama-papers/
“how many incidents of misconduct retractions have you covered and how many of these have been due, directly, to unethical actions by a pharmaceutical company? I continue to be curious over the seeming imbalance in checks and balances for commercially vs non-commercially sponsored research. I work in both environments and believe that nobody occupies the moral high ground (despite popular media assertions).”
Thanks for the reminder about this question.
The short answer is that we can’t recall any retractions due to misconduct that have been due “directly, to unethical actions by a pharmaceutical company.” But to use that as evidence that there is more or less misconduct in a particular sector would be cherry-picking the data in a highly misleading way.
For one, do drug and biotechs publish nearly as often as others? There are a number of recent studies showing that they don’t publish negative studies, and often they want to maintain their trade secrets. In an example of the latter, here’s one case of a retraction by two company employees who published without permission:
http://www.retractionwatch.com/2011/09/30/authors-retract-chemistry-paper-after-failing-to-get-companys-permission-to-publish/
Another example:
http://www.retractionwatch.com/2011/07/15/cryptic-legal-issues-lead-to-retraction-of-alcohol-abuse-paper/
The relevant question, hinted at by the sentence that follows Dr. Woolley’s actual question is, how many retractions of commercially sponsored research we’ve covered, whether due to proven misconduct or not? The answer to that would be “many.”
In fact, the very post on which Dr. Woolley poses her question is an example of one:
http://www.retractionwatch.com/2011/09/07/anil-potti-failed-to-disclose-corporate-ties-in-yet-to-be-retracted-jama-papers/
Potti had corporate ties that he did not disclose to JAMA when he and his co-authors submitted their paper, now retracted:
http://www.retractionwatch.com/2012/01/07/potti-and-colleagues-retract-2008-jama-paper/
Another example: Joachim Boldt, who has had funding from several drug companies and has been forced to retract some 90 papers:
http://www.retractionwatch.com/category/by-author/joachim-boldt-retractions/
In the end, we agree with Dr. Woolley — “nobody occupies the moral high ground” — and we’ve never said anyone does.
Another country heard from. Fascinating, and you are right: a pharmaceutical company would have to be suicidal to intentionally sponsor fraudulent research. For the simple reason that, if they were to market a drug for an indication only to discover that it didn’t work or was toxic because the research was flawed, they would face a significant financial loss from patient lawsuits. Such things are hard to cover up when many patients are harmed.
Fraudulent acts by researchers paid by pharmaceutical industry money are the personal responsibility of the individuals involved, and the industry is defrauded as well as the public. This is the same as fraud at a university or charity sponsored research center; the institution is involved only in the cover-up.
The motivation to commit fraud is usually based on laziness or incompetence and the need to cover it up; less often is it preconceived. The offer of money for positive results is the temptation. Wherever money is paid, the temptation will exist.
Thank heaven I am retired.
Excuse me, but when you have a record, commenting on the record is NOT ad hominism, but is simply noting a fact. Here is the specific question: Exactly how much money would you give Dr. Potti to conduct research? I would give him nothing, since his track record and past performance have led me to conclude that he is either very careless or simply imcompetent. That is not ad hominism, that is basing a judgement on data.
Can you explain to me why the Federal Government is not involved in this? Didn’t he have money from NIH? Isn’t it a felony to fabricate data on a federal grant?
They are involved. The Duke Chronicle has a good timeline here: http://dukechronicle.com/article/nci-questioned-potti-research-even-after-duke-review
Also, it hasn’t been proven that Potti actually fabricated data, and none of the notices refer to fabrications.
No, you are right and it was my mistake. I think I might have clicked on the wrong reply link. Your comment was indeed on the content of the paper. My comment was intended to be on the original article which I perceived as “oh my god, Potti published again – how dare he?”. This is ad hominem since not a word about the actual paper is lost. Your comment, though, is absolutely about the paper itself and I thank you for it. It would be great to have this kind of content featured in the actual articles of this blog so it does not become a site for gossipping about people. (Especially since I don’t know Potti in person so gossip about him isn’t particularly interesting to me. 😉
But what gives me pause for thought is that after his considerable fall from grace and against the enormous weight of the “common wisdom” now directed against him (not least as in this post and the comments) …… he persists in trying to publish against what must be formidable opposition.
And succeeds in getting a paper published in a reputable Journal.
“Black and white are only limits of the innumerable shades of grey”.
And perhaps there is more to Anil Potti than appears….?
Are you suggesting that he has powerful but invisible friends? I’m just saying…
@Jon: It’s not a felony. The Office of Research Integrity (in the case for NIH grants), usually gets involved and debars the individual(s) from receiving grants from the government for a certain period of time.
But, in certain EGREGIOUS situations, the State District Attorney has charged the individual with submitting false information to the government (which is a felony) when the individual has submitted progress reports to the NIH with falsified or fabricated data, but not the initial submission.
@Paul Thompson and Conrad Seitz
The review that Anil published is predominately based on the work of others, not on the work of Anil’s that has been found to be irreproducible by Dr. Baggerly. Anil was responsible for an independent approach that utilized a series of cancer cells called the NCI-60 in order to predict drug response. Anil’s retracted publications (ie. Nat Med paper) state that based on sensitivity of these ~60 cell lines to drugs such as conventional chemotherapies, an accurate predictor of drug response can be generated. It is now evident that the source data in Anil’s Nat Med paper was modified in such a way that it generated a positive outcome. Specifically, samples were mislabeled, leading to a significant result, where no significance was found with appropriately labeled samples.
In contrast, the pathway-based predictors have been utilized and reproduced by many. The pathway-based signatures are completely distinct from the NCI-60 based approach, and are generated using primary cells in which a single pathway is activated. The gene expression changes that occur following activation of a single pathway component are then used to generate a pattern reflective of that pathway’s activation. While Anil and anyone else with access to a computer has access to the pathway-based signatures (as the data is publicly available on GEO), they were not developed by Anil.
It is important to ensure the details of this complicated case are accurate. Dr. Baggerly was right in the context that Anil’s research was irreproducible. This data includes Anil’s NCI-60 (Nat Med) and NEJM research, and those papers have been retracted. However, there was data mismanagement by one individual only. By failing to distinguish Anil’s alleged misconduct from the valid and reproducible work by other scientists at Duke (and other institutions) who utilize genomics to generate predictive signatures and who have never been accused of misconduct, is to act with baseless and uninformed prejudice.
Thank you for the information on details of the Potti affair.
I understand from what you write that Anil manipulated the identity of samples in order to achieve a positive result in a line of cells which did not spontaneously show this result.
Is this not (alleged) fraud? “Data mismanagement” is really another euphemism.
I am beginning to see Dr Potti as a very sick man.
His behavior should be, just as you say, no reflection on the other researchers at Duke and elsewhere who are using pathway-based signatures to get reproducible predictions.
Like the vast majority of researchers, who are working hard every day, assembling data the usual way.
I opine that this Anil is more delusional than I previously thought.
Wait a bit. What about J Nevins? According to @Paul Thompson, that one published several papers with Potti. Could he/she be involved in any way?