We’re pleased to share this guest post from Leonid Schneider, a cell biologist, science journalist and a prolific cartoonist whose work graces our Twitter profile and Facebook page. In it, Schneider argues for a new way to ensure accountability for publicly funded research.
It has become clear that scientific dishonesty is rarely sanctioned. In the worst case scenario, manipulated or fraudulent papers have to be retracted, yet scientists of a certain academic standing can weather retractions without much adverse effect on their careers or institutional budgets. Retraction Watch lists plenty of cases in which the full blame for retracted papers has been accepted by the first author, with the senior principal investigator (PI) continuing as nothing has happened. Cases of justice, like that of the HIV-vaccine fraudster Dong-Pyou Han, who has apparently accepted a plea bargain after losing his job at Iowa State University, which in turn had to forfeit part of a $14.5 million NIH grant to the lab in which he worked, are very rare.
The problem is that scientific misconduct is difficult to pinpoint as a crime in a legal sense, as it only indirectly benefits the perpetrator financially. Grant funding, deceitfully procured through research misconduct is indeed not “legally” embezzled, as it does actually flow into academic research, through salaries, reagents, equipment, consumables and institutional overhead. Additionally, and probably also consequently, institutional investigations into alleged misconduct usually run dry, and grant agencies hardly ever ask their money back. This however cannot remain the case, or academic science as it is meant to be will eventually cease to exist, simply because the public may lose interest in financing faulty and fraudulent research.
It is reasonable to assume that a funding agency, public or charitable, may indeed be rather interested in recovering the huge sums it has previously granted on false assumptions. After all, most grant proposals are constructed around a small number of central recent publications by the applicant. If the central paper is retracted, the justification for the positive proposal evaluation is theoretically gone as well. Here, I would argue that it is actually irrelevant whether the rationale for the retractions was data manipulations, ethical breaches, plagiarism, fraud or even an “honest mistake,” if such a thing actually even exists. The previously successful funding proposal turned out to lack any foundation.
So why don’t granting agencies simply claim their money back? Because, at present, they have little chance of succeeding. The fractions awarded as direct salaries to PIs are either too small or legally hardly accessible for such damage claims, which can easily go into hundreds of thousands and even millions of dollars. Lab heads rarely keep their funding stashed in a bank account waiting to be raided by the furious grant giver. Instead, the money is already invested and consumed in the institutional research and part of the institutional budget. And this is exactly where I think change is needed.
The academic institutions, namely universities as well as research centers and hospitals, are largely co-responsible for everything going on under their roofs. Just as the institutions readily claim their chunk of fame and cash showers when their labs publish a high-impact paper, so they must also be made to share the responsibilities when such paper is retracted. These institutions must be made to repay the grants their researchers have dishonestly acquired.
How to do this? I propose that funding agencies demand a refund clause in every grant application, signed by the applicant’s current or potential host institution. This clause would specify the institutional responsibility in case the earlier publications by the applicant on which the proposal is based should ever be retracted, regardless of the reasons. Grant proposals without such signed clauses would not be processed upon submission. The percentage of refund can be further specified to consider the reasons for retraction, degree of how each cited publication by the applicant is to be weighted, the extent and transparency of the institutional investigation, the perpetrators’ stance, recidivism and so on.
Importantly, the clause should apply to every retraction, not just those in which misconduct is proven. Otherwise, the institutions would have an incentive to cover up misconduct, simply to avoid being fined for it. Even in the very rare cases of artifacts, initially misinterpreted in good faith, what would be the point of funding a further investigation into these artifacts as originally proposed in the grant application? This may appear unfair to the unlucky honest authors of artifactual papers, but supported by a thorough investigation, any serious grant agency would be inclined to find a compromise solution.
The benefits of a refund clause are many. Primarily, the repaid grant money will become again available to the more honest applicants, those who were left empty-handed back in the previous rounds. This is particularly important in today’s climate of austerity and budget cuts. But the side effects will be even more substantial. What faculty or board of directors will still support and protect a PI who caused their (usually notoriously cash-strapped) institutions to face such a hefty fine? The days of the old one-hand-washes-the-other culture, in which dishonest scientists could continue as before, with the silent blessing from their colleagues and employers, will be over. No promise of future papers will be that appealing to the host institute if it has to fear another retraction and the ensuing damage claims from the funding agencies. Even established professors and senior PIs, when caught in misconduct, may then easily find themselves out of a tenured job or at least have their lab stripped of assets, personnel and financial support. When filling new positions and at tenure hearings, the institutions may start considering factors other than just the impact factors and associated likelihood of grant funding.
On the other hand, PIs will be more mindful which kind of research attitude they encourage in their labs. Nowadays, some office-bound PIs hardly know or actually want to know how exactly the experimental results were obtained, as long as they fit the PI’s predictions and expectations. Thus, even a retraction can be shrugged off with a hint to an allegedly criminally-minded PhD student or postdoc, a snake at one’s honest bosom. Yet it will not be that simple when your employer comes after your tenure and your lab, waving a grant agency’s spectacular bill. Group leaders’ dreams of a “productive” junior scientist, who always delivers the experimental data nicely fitting the set expectations, may turn into a nightmare. Modern scientific research could finally move towards open-hypothesis and curiosity-driven approach, instead being the product of not always reliable or reproducible intelligent design of its creators.
Finally, large and influential grant agencies, when seeking financial compensation where misconduct is suspected, may also exert pressure on journals, many of which seem rather reluctant to retract papers even where manipulations are blatantly evident. The fact is, grant agencies like NIH, MRC or DFG (who also subsidize journal subscriptions) do wield a lot more power than Retraction Watch or PubPeer, in regard to dealing with the science publishers. Thus, even more of the faulty and irreproducible papers may get the axe, instead of being saved by editors and dishonest authors, who chuckle behind our backs by ignoring and dismissing the accusations or publishing inappropriate errata and corrigenda.
With such a solid civil law standing for refund claims, grant agencies should see as much easier to win in court, and it would be pointless for institutions to help PIs feign “honest errors” or shift the blame on some junior scientists. Refund claims for proven scientific misconduct and criminal investigations would remain unaffected. This paradigm shift may initiate a more profound change in research attitudes and activities than anything else.
There may be initial negatively perceived side effects, especially if a compensation claim-slapped institution decides to cut down on all research labs inside its walls, both bad and good. Yet keeping dishonest scientists in their jobs and power may be even more detrimental to the financial situation of their honest colleagues. Moreover, why not introduce a rebate system for institutions, provided these can credibly live up to their promise of a misconduct investigation, especially if the universities and research centers fix the misconduct issues convincingly and restore scientific good practice before the compensation claims from the grant agencies even arrive? The fines may not even have to be enforced, but the science and the papers the institution produces would still become very different. Certainly they would become more reliable.
TL;DR, I must say. However, I think I get the gist.
I don’t disagree with the premise. However, institutions are the ones that investigate research fraud and they sometimes seem to sweep it under the rug. Holding educational institutions financially responsible when fraud is proven creates an even greater incentive for institutions to sweep it under the rug.
Ultimately, it should be the researcher who is held accountable. It should be treated no differently than fraud in any other field. (i.e., fines and prison time).
exactly the thing I would like to see avoided, see my 7th paragraph:
“Importantly, the clause should apply to every retraction, not just those in which misconduct is proven. Otherwise, the institutions would have an incentive to cover up misconduct, simply to avoid being fined for it. Even in the very rare cases of artifacts, initially misinterpreted in good faith, what would be the point of funding a further investigation into these artifacts as originally proposed in the grant application? This may appear unfair to the unlucky honest authors of artifactual papers, but supported by a thorough investigation, any serious grant agency would be inclined to find a compromise solution.”
Even today, journals sometimes have trouble with correcting the scientific records because they are being denied access to the reports and records from internal investigations. If the host institutions have a direct financial incentive to avoid retractions, then there will be no more retractions. It’s as simple as that.
Bernd, if the journal finds image manipulations and the authors refuse explaining, the paper is always pulled. Even nowadays. The problem is, authors and institutions do reply and manage to sweet-talk the journals, by pretending to find the correct data, and confirming the results and so on. All of this never becomes public, and the grant agencies are never involved
That is what I was going to say. You hit the nail on the head. The universities are the fox guarding the hen house. The only way that reform will work is to institute an independent body that will look into these matters.
One thing I did not like in the piece was the suggestion that there may not be such a thing as an honest mistake. Let’s not get carried away. Mistakes can happen. I fear that when people finally realize how big the problem is we will go overboard the other way.
Indeed, I was more thinking about life sciences, were honest mistake is likely to be euphemism for negligence. Than expression of mine was not appropriate here, and leads into the wrong direction.
Awards aren’t given in the name of the researcher; they go to the institution. If institutions, which self-investigate, have to give money back with a finding, doesn’t that pose an even greater COI?
There needs to be greater enforcement both by ORI and OHRP in general.
The more transparently the institutions investigate, the less they have to pay back, that’s the idea.
The university, journal publishers, and government funders are one mutual benefit political system. They all are a product of groupthink gamers from top universities. Who in this system really wants to stop system gaming? Every bureaucratic system starts with lofty goals, only to be corrupted by greed and groupidity. That is where the NIH is today:the biggest tax scam ever devised, too big to fail to big to change,
Your proposal is a recipe for bureaucracy. Every grant application would require independent verification, all papers from an institute would require Good Laboratory Practice – like authorisation. (If you don’t know what that is check http://www.mhra.gov.uk/Howweregulate/Medicines/Inspectionandstandards/GoodLaboratoryPractice/Structure/). GLP requires regulatory authority compliance monitoring and very detailed documentation of all experimental work (for example, the protocol or work plan for experiments would require to be checked before the work started and then the results audited for compliance with the protocol when the report was completed; any change in the conduct of the experiment from the authorised protocol would have to be verified and the reason for the change noted, all scientists would have to have records of their training in scientific method and scientific instruments etc. etc.). It would move responsibility from the scientist to the organisation.
It has always been claimed that the strength of the scientific method is that repetition by others verifies the conclusions. Responsibility for the conduct of scientific work rests with the author(s).
I think that implementing this proposal would be very expensive and would stifle innovation.
The idea is not at all to supervise scientists, they can do it themselves. The clause would only spring into action when the paper was retracted.
jeanachete – Interesting comment. Not sure whether to take it as a tongue-in-cheek comment or a serious one. The principles of GLP simply codify good scientific practice (document what you do, have a clear plan, know the quality of what you are using etc – doveryai no proveryai) with an oversight by a QA unit. I certainly hope that all academic labs work under the “spirit” of GLP. The “laissez-faire” approach that you imply does help me understand why some grad students are so unprepared for life in industry. I really do hope that your comment was a tongue-in-cheek comment.
My comments were not tongue-in-cheek.
My reasoning is this: In a situation where a large amount of the grant money has to be returned to the granting body, there are bound to be cases where the issue ends up in court. The institution will then have to defend its record of care in relation to the scientist whose article has been retracted. This means that claiming the ‘spirit’ of GLP was followed will fail when the detail of that particular fraud (or error) is examined. It won’t be good enough to say:’those images, which were incorrect due to fraud, were produced under the spirit of GLP.’ Once an institute has had to pay back a large grant, it will want to ensure it doesn’t happen again. So when the court examines the detail of a particular claim, the institution will want to be able to defend itself by having suitable records showing it has pursued the correct compliance policy.
By the way, I have worked in industry and in the university system. To claim that the academic institutions work to the spirit of GLP, and thus there is not much difference is not correct. I wonder if you have ever worked under GLP? GLP imposes constraints on scientists which make it difficult for them to be innovative. Its OK if you are following a method which is well established; if the method is novel it is a different story.
Leonid’s concept is excellent. Currently those who fake research, and their institutions, have almost nothing to lose, and much to gain. This leads to moral hazard (http://eurheartj.oxfordjournals.org/content/35/36/2433.full-text.pdf). When crime pays on average, crime becomes a rational choice.
There is only one improvement I can suggest to his plan for funders to recoup their expenditure. Currently institutions are in charge of conducting investigations. This is already difficult.
– Some institutions conduct thorough enquiries and report them, but find themselves unable to clearly state that there has been research fraud. In one case there are multiple impossible features in a landmark clinical trial (www.bmj.com/content/349/bmj.g5210.full.pdf), but the institution has not specifically asked the two journals to retract the papers, and therefore the two journals consider themselves unable to issue retractions.
– Sometimes institutions embark on prolonged processes, lasting many years (http://blogs.nature.com/news/2014/02/evidence-of-misconduct-cardiologist.html). Often the big picture is more readily seen by those at a distance (http://www.sciencedirect.com/science/article/pii/S0167527313008012/pdfft?md5=1e1934216e01043d4d5dbcf6d651613f&pid=1-s2.0-S0167527313008012-main.pdf).
I am concerned that creating another adverse consequence for institutions of successful completion of misconduct investigations may slow them down or prevent them even starting.
Suggestions for taking Leonid’s suggestion up to the next level?
– Right for the funder to obtain all the raw data at any stage, and answers to any reasonable questions.
– Obligation for institution to hand over all data and answer any question asked by the funder, without delay.
– Right for the funder to retract the paper if these data are not satisfactory or if its questions are not answered adequately and speedily.
As for the question of how the university is to find the $$ to return if it has already been spent? How about the funder requiring it to have insurance that pays out when this happens. The insurers will work out a way of judging whether the scientists can be trusted, and charge accordingly. Those whose research often turns out not to be replicatable by others may still find themselves extolled by their flunkies as geniuses, but would be recognised by the insurance market as walking dynamite.
Suppose you had to quote your retraction/irreproducibility risk premium at the time of grant application for the funder to pay it. The peer reviewers could see it and might view the application accordingly.
People might think twice before fiddling the figures, or accepting numerous guest authorships.
We’ve heard of credit ratings; this would be a scientific credibility rating.
Darrel, many thanks for your constructive comment. I fully agree, but your third point “Right for the funder to retract the paper if these data are not satisfactory or if its questions are not answered adequately and speedily” may be ignored by journals like Nature who demand that all authors agree to retraction.
As for costs, how man retractions does an average university actually face annually? What is the grant funding affected? It is not like RW reports hundreds of retractions daily in US alone. I think it would still be manageable (even without insurance!). But the risk o fa retraction would change the attitudes.
I agree that more severe penalties for senior investigators who commit research misconduct could have a deterrent effect on individuals. Were the institutions which received both the direct and substantial indirect costs required to return those funds, it is also likely to prompt greater institutional attention to scientific integrity,.
The good news is that there is no need for new legislation To submit false data on an NIH/NSF grant application is fraud, a Federal crime punishable by jail time and/or a fine.The problem is not the lack of a law, it’s failure of the office of the Attorney General to prosecute offenders.
Unless it occurred in Iowa.Senator Grassley recently focused attention on a case that eventually involved prosecution in the case of a researcher from Iowa State University. The individual has pleaded guilty and awaits sentencing; NIH has rescinded1.4 million dollars from the individual’s research team. ( Des Moines Register JULY 8, ’14).
What is needed is more pressure from Congress on the Attorney Geneal to pursue these cases more vigorously. A they say in politics,Call your Congressman and tell him how this important issue is being ignored.
Don
this is why I want to move it away from criminal courts to civil ones. Criminal intent in scientific misconduct is very tough to prove, people are very rarely ever found guilty, and usually by their own admission. Turning a grant into a contract of data reliability would make a court case relatively simple.
If universities had to agree to refund grants whenever there was a retraction, I think there would be fewer retractions. Not because of a crackdown on misconduct or sloppy science. But because universities would try very hard to avoid ever allowing a retraction.
In the case of honest mistakes, universities would have a financial incentive to strongly discourage their scientists from doing the right thing and retracting flawed research. They would argue for corrections instead – or for publishing a new paper explaining the mistakes of the older one. Anything to avoid a fine. It would be a strong-minded scientist indeed who argued against the advice of their own administrators that a costly retraction was justified.
In the case of deliberate misconduct, universities already have plenty of incentives to brush a problem under the carpet. Now they would have one more financial incentive to underplay the seriousness of an offence. If they did admit that a paper lay on shaky ground, then, again, they would likely argue for a correction, or an erratum, or another paper, or just about anything short of a retraction, which would cost them money. While journal editors, funding agencies and investigatory bodies (like the ORI) could steamroller over those arguments regardless and demand a retraction, they’d then lay themselves open to counter-suing from the university. It could get messy very fast.
Could the funder build it into the contract?
“If we give you this free money, we have the right to obtain all raw data (gels, anonymised patient databases, etc) and any information we want regarding any paper which you publish with it, and can instruct a journal to retract the paper without your ability to veto, if we consider it advisable in the interests of scientific integrity. You are welcome to put an explanatory note after the retraction.”
So pharmaceutical companies will have an even easier time to whip researchers into submission, so they’ll come up with the desired results?
It is the question who is more powerful, the institution or the wronged funding agency.
I agree that invoking the “refund” principle for all retractions is much too broad,
Prosecutions for fraud should be applied to individuals who have been found guilty by ORI and have committed serious acts of misconduct. Such prosecutions are now rare.
As I have previously stated, i believe that such well publicized cases could have a chilling effect on those contemplating research misconduct. It might also motivate institutions to strengthen or initiate policies designed to prevent research misconduct.
Don .
I think readers of Retraction Watch may sometimes see misconduct as a general practice, rather than the typical assumption that researchers act on good faith. Punishment for fraudsters, sure. However, I worry about pulling out a bazooka to kill a fly. It may have a chilling effect where institutions may be less open to granting more ‘outside the box’ research that may be more subject to controversy.
Re: “It may have a chilling effect where institutions may be less open to granting more ‘outside the box’ research that may be more subject to controversy”, I have formed the impression (from what I have read here, only over a few months, and without any expertise whatever in the fields which seem to have the most retractions) that “being retracted” and “being outside the box” are, if anything, negatively correlated. Does anyone have actual figures on this? (Not that it’s clear how to quantify “being outside the box”, particularly as “the box” may move with time, and some of its movement may be tied to research that retrospectively is found to be more or less unreliable for any of several reasons.)
Three things will be coming out of this idea if implemented: 1) yet another layer of useless bureaucracy will be created that will be syphoning the very funds that you want to recover, 2) additional regulatory burden will be imposed that will require additional offices and bureaucrats to produce and implement these regulations who will be syphoning the very funds you are trying to recover 2a) to counter potential misconducts by said bureaucrats a regulated appeal process that requires special Office of Appeal will be created and staffed by yet another layer of bureaucrats and finally 3) the number of pointless, low-risk, incremental grant applications will increase. In the end this idea, as any other regulation-based idea, will produce a mouse, a very expensive mouse that contributes absolutely nothing to the quality of research, decreases funds availability, curtails academic freedom even further and increases the role non-scientific administration in determining what and how should be studied. Thank you, but no thank you.
Science, by the nature of the endeavor is a risky business: there is a risk that you might be wrong (in fact most of the time), there is a risk that you need to adjust direction of your research mid-project, there is a risk that you might commit fraud. The idea that people publish genuine artifacts rarely is, shall we say, hardly supported by any evidence. I can think of several Nobel Laureates who published results that eventually turned out to be incorrect and were later refuted.
In essence, research funding is no different than venture capital investment. Any serious venture capitalist will tell you that it is pretty good investment outcome when 20% of funded projects become profitable. With respect to academic research funding, I hope that we are nowhere near the 80% fail (fraud) rate.
The retraction risk for an honest, careful and diligent scientist is: practically zero. It doesn’t matter if you are inside or “outside a box”, as long as you do science properly, your paper will be safe. Even it gets contradicted or even proven wrong. No paper ever got retracted for its right or wrong conclusions, but solely for its faulty data.
I don’t think we know this. Ivan here actually had statistics at some point. As far as I remember only ~30-40% of retracted papers were retracted because of outright fraud.
Are you sure you really know when it is appropriate to ask a journal for issuing a retraction? See http://retractionwatch.com/category/doing-the-right-thing/ for many, many cases where people have asked to retract their articles for reasons that do not involve fraud.
Bernd, as I keep saying: the goal is not to reclaim all the cash, but 1) to increase the risk for manipulating data in publications and 2) to give the grant agencies the tool to force institutions to collaborate on a transparent investigation, otherwise they will have to pay everything back in full.
“there is a risk that you might commit fraud.”
Uh, no. By definition, fraud is something done intentionally. There is no risk of it unless you choose to do it. There’s a huge difference between committing fraud and being wrong.
Perhaps I was not clear here. From a funding agency point of view its is “risk” that they simply have to absorb. Same as, for example a risk of PI being hit by a track and therefore unable to complete project. Basically, any “agency” that invests in science operates with certain amount of risk of not getting what they are paying for. Fraud is simply an element of this risk. The question here is whether creating a whole new structure to deal with relatively minor problem is cost-effective. In other words does it make any sense to spend $20 millions to recover $2 millions as opposed to invest these $20 millions as a risk of losing additional $2 millions to fraud. A priori the answer is not obvious because we don’t know how much funding gets wasted to pure fraud.
Actually, fraud is much more common that one thinks, but semi-fraud is much more widespread and dangerous. It rarely leads to retractions, so for a semi-dishonest scientist the risk to get caught and face a retraction is negligible. So a refund clause would make this risk so high that it may become intolerable for the institution to employ and protect scientists whom it doesn’t sufficiently trust.
Does this “much more common” has a number associated with it? Because if it does not, I am sorry, but it is hand waving.
there are none and there unlikely to ever be any reliable statistics for misconduct in science, for the very obvious reasons. But feel free to talk to any scientist you meet. If you wish to go for court proven cases as your statistical template, than indeed I agree, there is hardly any misconduct problem at all and we all should relax and trust the scientists.
Wouldn’t proven fraud and/or abuse that is part of the alleged research misconduct fall under the False Claims Act, if federal funding is involved?
Of course it would, Ned. The Federal Government (Department of Justice) or individuals can file FCA suits to try to recover misspent federal funds, with up to triple damages. Relatively few individual suits are successful though, and DoJ is reluctant to pursue them without a solid case with substantial return (over $1 M)
So, according to this proposal, money will be retrieved and sent back to the same individuals who created the same system that doled out the tax-payers’ money to undeserving scientists? This has no chance ever of working, and should never be implemented. At least it does not have my stamp of approval (as a tax payer). More wasted funds on bureacracy. More poverty for the tax-payers. Funny, whenever the real tax-payers talk about tax-payers’ money, they don’t ever really get anything back, in real dimes and cents. Only their governments benefit, and the lawyers, too. No, we need a more fundamental change of the system. Admittedly, a retraction is not enough: it is like the public alarm-bell, nothing more. Penalties and criminal prosecution, that is what is required. Instead of funding government behemoths like ORI, with no legal teeth, scrap the money being wasted on ORI, and hire a small team of top-notch lawyers who will go after the money wasted by cheating scientists. Let the lawyers get their cut of the recovered funds (incentive?) and make sure that at least 50% of what is recovered goes BACK into a special fund that honest scientists with real projects can vie for. Whatever scheme you create, make sure that the government does not get 100% of the tax-payer’s money.
there is not much to get from scientists’ own pockets, especially when a paper has 20 authors all of whom plead innocence and without someone pleading guilty nobody can ever prove in court who was actually the guilty party. Never mind how much the lawyers get paid.
There are about 350 malpractice retractions per year; let’s imagine 100 of them from US institutions. The fear of having to return a grant would likely create overhead/departments to prevent this at each place. It will likely be at least $100,000 per year for prevention. Then 3,000 US institutions will spend $300m in total each year to possibly prevent 50 retractions.
The question is always how to get the best return on each research dollar and how to increase reproducibility. Making universities pay back grants for retractions will waste countless taxpayer dollars. And as Richvard commented above, would also likely decrease retraction statements themselves.
Adam and Ivan have also written before about ways to increase accountability, by giving ORI more power (http://www.nytimes.com/2014/07/11/opinion/crack-down-on-scientific-fraudsters.html?_r=0). That seems reasonable and more appropriate than the above proposal.
The key problem for science is not malpractice and retractions but the non-reproducible research in many normal papers. There are many good things to do to increase reproducibility, for a much lower cost to society.
Lenny, as I suggested, the whole system will only work if EVERY retraction is followed by refund, regardless of reasons. There is no need to prove or disprove misconduct, unless the institution wishes to convince the grant office to look mildly upon the penitent perpetrator.
I like this goal–and even proposed it in some earlier RW comment–but think getting any federal agency to be more accountable will first take Congressional action, and that will take a lot more scandals and public outrage.
How about if RW and other science bloggers just start reporting how much grant funding was spent on a retracted study, and how much the authors and their institutions have refunded or paid in fines?
I can imagine a day when such matters are reported as part of journal retraction notices: “This research cost X Agency $A , of which $B has been or will be refunded by the authors or their institutions.”
Sunshine is a great disinfectant and raising this issue in every retraction notice would get more scientists used to the idea that this matters.
If not already planned, I hope RW will include fields in its new retraction database for the sources and amounts of funding associated with retracted papers as well as the amounts of refunds and fines.
RW could then make a cool widget or app like the Debt Clock that displays a running tab of how much money has been wasted on retracted research and how much recovered.
I’d happily pay $.99 for it. Especially if i could search these stats by author, institution and funding agency.
Albert, a clarification: getting money back which went into a retracted study is impossible. But claiming money back which was awarded based on a study which was later retracted, is a different issue. Authors themselves excessively stress the importance of their recent paper when applying for a grant, so there will be no need ever to prove the correlation.
This is a tremendously bad idea, for one reason alone: Universities would have to bank enormous amounts of money to pay up for retractions, sometimes years after the grant was ended. How can you manage a budget when unknown and completely unpredictable expenses could appear at any moment?
On second thought, a “retraction insurance” dodge could be lucrative. Could create jobs, boost the economy….
Indeed, good point! Universities have insurances for everything, so why not against retractions?
The insurance premiums will have to be paid out of the grant one way or another, so you’ll have the honest ones bearing the costs, while the fraudsters get a free ride as the insurance will pay for them.
Bernd, as I said above: As for costs, how man retractions does an average university actually face annually? What is the grant funding affected? It is not like RW reports hundreds of retractions daily in US alone. I think it would still be manageable (even without insurance!). Again, it is not about the money, but about changing the institutional attitudes.
Not even close. See “The Military Industrial Complex,” which yearly in the US alone absorbs approximately $750 billion in tax dollars. And if you think that all goes into bullets and guns, guess again: the 2013 salary for Marillyn Hewson (CEO of Lockheed Martin) was >$25 million, for W. James McNerney (CEO of Boeing) it was >$23 million, and for David Lesar (CEO of Halliburton) it was >$20 million [source: http://www.aflcio.org/Corporate-Watch/Paywatch-2014/100-Highest-Paid-CEOs ]. And that doesn’t even touch their “options,” “allowances,” “benefits,” “considerations,” etc etc.
This isn’t to say that NIH, NSF, and all the other funding (and investigatory) agencies of the US government do not need to do a far, far better job of recouping taxpayers’ money in cases of fraud. They certainly do. But compared to the long-running scam mentioned above – and that’s only the barest tip of the iceberg – NIH is still a good deal for US taxpayers. Referring to the NIH as “the biggest tax scam ever devised” is…well, full of sound and fury. You know the rest.
You are right about the military/industrial complex being a bigger waste than the NIH. So I guess your logic is that is that being bad is not as bad as being bigger bad. We should expect better, not compare as being less bad.
A grant or fellowship is assistance funding – not a contract (where such requirements might be included).
Richard, Kevin and YC above are correct – the proposal would cost an additional administrative and legal complexity for the agency, far more cost than any recovered funds.
it is less about the money itself, but as an incentive to be honest. As I indicated, the prime goal for a grant agency is to enforce a transparent investigation by offering a rebate, as opposed to dumbly demanding money back.
You may have missed my earlier comment.
There is no need to create new legislation or a new enforcement agency
Under current law,to submit fraudulent data in a Federal grant application is considered to be fraud and is subject to prosecution by the Attorney General
But the trouble is that many (I think most) papers are retracted only because either the authors or their institution cooperated with an investigation started by a journal. Without the help from an institution, there can be no access to records, physical evidence and human resources (a university can silence its employees easily enough).
It is a bold journal that retracts without solid evidence, and the institution is the key to that evidence. If your plan gives institutions an incentive to not cooperate, then that will only result in far fewer retractions.
Actually, if authors refuse to cooperate, the paper gets pulled. This is why they throw sand (look, we found the correct data! but the findings are nevertheless valid!) into editors’ eyes to avoid retraction.
“I propose that funding agencies demand a refund clause in every grant application, signed by the applicant’s current or potential host institution. This clause would specify the institutional responsibility in case the earlier publications by the applicant on which the proposal is based should ever be retracted, regardless of the reasons.”
Instead of a “refund clause,” how about if the Department of Health and Human Services and/or NSF and/or some other federal agency (National Science Board?) formally license institutions that receive research grants? In response to a violation of the license terms (e.g., research fraud or misconduct revealed by an ORI investigation) the licensing agency can implement appropriate sanctions, which might include fines or license revocations.
The model for this is the US Department of Agriculture. If I understand things correctly (and I certainly might not) it issues licenses for animal housing facilities, including those at research institutions that may or may not receive Ag department funds. If there is *any* violation, regardless of the underlying causes and circumstances, the Ag department can levy sanctions that include fines (for example, http://www.bostonglobe.com/metro/2013/12/18/harvard-fined-for-violating-animal-welfare-law-care-monkeys/5NZT6eld8fR1V6tyuRxOWP/story.html).
There needn’t be a new, mega bureaucracy created to oversee such license compliance – the Ag department does it itself, so why not NIH or NSF? Or perhaps ORI? The Ag department investigates *and* imposes fines as it deems appropriate, so in principle ORI should, with relatively minimal extra expense or bureaucracy, be able to act similarly.
Good idea, the final result would be the same: grant agencies could fine institutions for retractions as violations of licence contract.
That’s correct, or not fine offenders if the circumstances warrant – in those cases, for example, of ‘honest’ (if sometimes careless) mistakes. This model would place additional responsibility upon ORI, but who/what outside of Retraction Watch readers would be better positioning for determining appropriate sanctions? And if the licensing agency functions with budgetary and personnel independence from NIH, NSF, DARPA, ONR, etc., potential conflicts or interest would be minimized.
Of course there’s always the possibility of corruption: in an effort to retain his multi-million-dollar pot of indirect costs, President Poobah of Shaky State University was seen slipping c-notes into the pocket of an investigator. But that’s a real risk for any investigation performed by any agency or individual with sanctioning powers. We would just need to trust the system, the same way we trust scientists…uh…
Actually, if an investigation is indeed transparent (unlike most investigations into misconduct etc so far), there is much less room for corruption.
This sounds like a bad idea. What is the phrase, “throwing the baby out with the bathwater”?
This is a dreadful idea. I can only assume Schneider wrote this tongue-in-cheek. Firstly, imagine the lawyering-up required. Nobody wants to see that, especially not in academia. Secondly, groups will fight tooth-and-nail to avoid retracting a study. As a result, far fewer (suspect) papers will be retracted.
Ed, as it is now, groups do fight tooth-and-nail to avoid retracting a study. The key problem is that the institutions will do everything to cover up and to protect dishonest scientists as long as those are still able to bring grant funding. Alternative ideas to break this are welcome.
Interestingly, when applying for Grants, the research does necessarily have to out right lie or commit fraud, they simply leave out the negatives on their research, then exaggerate and embellish on what appear to be the positives. Once Grant Funds are in hand, it’s easy riding from here on out.
Leonid Schneider comments “even an “honest mistake,” if such a thing actually even exists”? In these very pages a few months ago we read about Lance Fortnow, at http://retractionwatch.com/2014/12/15/dont-retract-paper-retract-results-within-one-scientist-still-displays-one-mistakes/ . He published a proof in 1988 “then discovered “unexpected technical challenges” and published a retraction in 1997.”
We read about how “Authors retract two spectroscopy papers when follow-up results don’t match” http://retractionwatch.com/2014/10/28/authors-retract-two-spectroscopy-papers-when-follow-up-results-dont-match/ and where the only commenter wrote “From what I can tell, this group did the right thing and pulled some misinterpreted results. Why exactly is this a story?”
We read about the “two retractions by Pamela Ronald and colleagues, after the group found that a bacterial strain they’d been using was contaminated” http://retractionwatch.com/2014/01/14/following-up-pamela-ronald-publishes-updated-data-following-two-retractions/ and their “paper in PeerJ following their investigation into what went wrong”.
We read about David Vaux, at http://retractionwatch.com/2013/06/19/why-i-retracted-my-nature-paper-a-guest-post-from-david-vaux-about-correcting-the-scientific-record/ , who “wished to retract [his] News and Views piece because I no longer had confidence in the findings on which it was based.”
We also know from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3479492/?tool=pubmed that of the analyzed retractions, “21.3% of retractions were attributable to error” and “67.4% of retractions were attributable to misconduct.”
What more evidence is needed to show that honest mistakes occur, and with a high enough frequency that the possibility has to be factored in? Why have Retraction Watch if we don’t learn from it?
Andrew, all these cases and their statistics are based on authors’ self-assessment, which is by definition subjective. Without a transparent investigation, authors claim for an “honest mistake” cannot be taken at face value.
What evidence do you have that that is the reason between the PNAS paper’s finding that “21.3% of retractions were attributable to error” and your statement that these are “very rare” is due to self-assessment writings that the PNAS author’s couldn’t identify? Why should I believe you when you say these are very rare when I can find many counter-examples of people who issued a retraction and seemingly without fraud?
Look in the category of “doing the right thing” at http://retractionwatch.com/category/doing-the-right-thing/ . You’ll see three physicists who had a bug in their Monte Carlo program, at http://retractionwatch.com/2014/10/03/doing-the-right-thing-particle-physicists-pull-paper-after-equation-collides-with-the-truth/ . Your view seems to be that that’s very likely fraud, and we’ll never know unless there’s a transparent investigation.
Why can’t you believe that people make programming errors and fail to catch it until after publication? Geoffrey Chang’s infamous sign error, and retractions of 3 Science articles, a Nature article, a PNAS article and a JMB article, is clear precedent.
As http://retractionwatch.com/2013/08/29/doing-the-right-thing-researchers-retract-two-studies-when-they-realize-they-misinterpreted-data/ points out, “What do you do when new experiments show that you interpreted the data from your old experiments the wrong way? Some scientists might just shrug and sweep those errors — and their previous papers — under the rug.” Why would anyone subject themselves to investigation when they can just ignore the problem? After all, if they are going to be considered a fraud either way if everyone were to think like you.
You’ll see another example of a piece which was retracted because it depended on the result of another paper which was retracted, at http://retractionwatch.com/2014/09/25/downstream-effects-comment-on-retracted-narcolepsy-paper-retracted/ . Your view seems to be that (like Vaux’s News and View piece I mentioned earlier) downstream retractions are automatically considered a fraud, trigger an investigation, and require returning any grant money.
Those are absurd views.
Leonid, I don’t think that can rightly be said of the Fortnow case, or of mathematical substance-based retractions—as contrasted with retractions due to discovered plagiarism (for instance)—in general. Mathematics (including the kind of theoretical computer science that is Fortnow’s speciality) seems to be quite different from the sciences (even, I think, theoretical and mathematical physics) in several relevant ways. Jody Azzouni’s account of “How and why mathematics is unique as a social practice”, reprinted in R. Hersh (Ed.), 18 Unconventional Essays on the Nature of Mathematics (pp. 201–219), is very convincing (to me). Here’s a brief quotation (I hope it survives my cut-and-paste; his italics):
With the exception of (rare!) pathologies (Smarandache has been mentioned here…), if a blunder goes undetected it’s because its purported result hasn’t been seriously used by anyone (even its own author), and when someone (maybe the author…) does come to use the result and find the blunder, it gets corrected (if possible) or withdrawn (if not); and in either case, what usually (not always) happens is that more mathematics results! Azzouni uses the phrase “the benign fixation of mathematical practice” for this process.
I can give a personal example. I’m nearing the end (I hope) of preparing a long paper (on “constructions and examples of high-dimensional C-links”) that both generalizes and depends upon 40 years of work by many people, including me (on, what else?, “low-dimensional C-links”, mostly called something else). In the process, I have discovered an oversight in a paper I published in 2001 (essentially, I asserted that a certain sufficient condition was both necessary and sufficient, but actually it is NOT necessary). In this case, nobody else (including the—very thorough, as I recall—original referee) noticed the oversight, but it hasn’t caused any trouble because until now no one has used it (in contrast to several other ideas and results from the paper which have been used, and usefully generalized, by the authors of at least 14 papers, 5 of them since 1910). If someone else had tried to use the “necessary” part of my statement, she or he would have noticed. As it is, I have found a dandy (and this time, correct) NASC statement; my new paper will admit the old error (but I don’t see any need, or any channel, to retract the old paper), and mathematics will be no worse the wear.
Is this a genuine comment or sarcasm? I’m struggling to see how it could be true.
The COPE guidelines state :
“Retraction should usually be reserved for publications that are so seriously fawed (for whatever reason) that their fndings or conclusions should not be relied upon.” ( http://publicationethics.org/files/retraction%20guidelines.pdf , p. 3)
The reasons can include “clear evidence” of unreliable findings (due to misconduct or honest error – yes, it gives examples of honest errors), “redundant publication”, plagiarism, and unethical research.
Yet on the RW site there are retraction notices citing possible trademark infringements (eg EpiPen), non – payment of open access fees, disputes between co – authors, and mistakes by publishers (including somehow managing to publish the same paper twice in the same journal).
ELF, as long as the data presented is not flawed, its theoretical interpretation cannot be subject for a retraction. In rare cases, where artefacts lead to retraction, an unspecific staining was presented as a specific one, which made data unreliable.
I hope that you are all aware that it is now possible (and has been done) for funding agencies to recover funding which went into producing plagiarized papers or otherwise flawed ones as part of the funded research? This is all without any additional rules or governing bodies. When the final report is submitted, someone actually reads it. If they find plagiarism or other scientific misconduct, they can insist on money being returned. This does not necessarily involve published papers only, but the entire reporting process.
Thank you very much for pointing this out, Debora! So after all, my idea is not that radical, and may see even have a chance to be implemented.
Debora, it is not quite as easy for federal “funding agencies to recover funding which went into producing plagiarized papers or otherwise flawed ones as part of the funded research” that “If they find plagiarism or other scientific misconduct, they can insist on money being returned.”
In my experience at ORI, when NIH tried to get money back on such grants, NIH had to show that NIH funds were actually used for the plagiarized or falsified work [sometimes an institution could show that NIH funds actually were not used, despite the acknowledgement in a paper by the PI].
And NIH had to determine what fraction of the grant was affected by the research misconduct (NIH could not just assume that the whole grant was affected and that all the money should be returned).
Retractions are symptoms of scientific fraud. Whether fraud is committed is an other issue. Whether that fraud results in an automatic recovery of grants by a funding agency either from the person himself or his institution is a second issue.
Compare this with the case of a staff member in a company. If he commits fraud, eg collects or pays bribes, there are consequences for him and his company. I think that existing legal frameworks covering fraud should be used, which firstly defines what is fraud, and secondly – between individual and institution – apportion retribution, repayment or punishment according to existing principles, which might vary from country to country.
So an automatic recovery of funds – as proposed above – is not right. By using existing legal frameworks, one can reduce the extra overhead that one rightly complains about.
An interesting issue is, whether scientists should fulfill higher ethical standards than others as they should be examples in impartiality and honesty. Just as much as one expects that police and priests are examples of good citizens.
I like the idea. The dangers of it leading to new layers of bureaucracy are overstated. However, the idea can only work in the context of “open data” and in this case the benefits will be very substantial.
I wrote a full comment, that got so long that I posted it here
https://ferniglab.wordpress.com/2015/01/21/should-universities-refund-grants-based-on-retracted-papers/
if anyone has the inclination to read the detail.
Many thanks Dave, i very much value your opinion. Of course Open Data goes without saying. But even subscription journals cannot keep hiding things, with or without subscription fee, because they are published, as in PUBLIC.
I am sorry I was not able to explain this more clearly.
1. The goal is to stimulate investigations and transparency, using a threat of a hefty fine. A bit similar to tax evasion: timely and cooperative self-reporting results to a massive reduction in punishment. Most importantly, not the repayment of the total fine is set automatically, but the right of the grant agency to claim it. And grant agencies are no loan sharks, they are more interested in supporting honest and reliable science than certain other parties.
2. It is not the institutions who decide upon retractions, but journals alone (coordinated with authors). All the institutions can do there, is to either press the authors to retracts or obfuscate. If the institutions obfuscate, and the journals retract anyway (because they receive the same anonymous hints as everybody else and also have their own tools) the grant agencies have all the rights to claim their money back in full. If the institutions instead cooperate and present a transparent investigations (e.g., determining all the real culprits or proving it was indeed an honest mistake), the grant agencies would reduce or even waive the fines.
3. My idea may appear radical. But proving personal misconduct in court or elsewhere hardly ever works. Leaving everything as it is will only benefit the dishonest scientists, who are protected by their institutions as long as they are liable to bring further grant money.
Putting aside the efficacy for a moment, I think the mechanisms for this are just as interesting. You could use models from other kinds of financing, for example putting some or all of the funder’s money into escrow. The university would pay all research costs as they come in, but as certain agreed-upon benchmarks are reached, the funder’s escrowed money would be released to the university. The key condition would be that a sizeable percentage of that funding could not be released until, say, a year after publication of results: an “ethics window”. If research misconduct is uncovered during that time, the final tranche of funds would return to the funder rather than released to the university.
I think a simple solution to this would be to implement something like what I have to go through with my commercial roofing company: surety bonds.
Even with only partial reimbursement of the total amount granted for the research, etc., this opens up a small secondary market for tracking the individual performance of departments and institutions, and individuals, with built in punishments over the long term for non performance – you lie or cheat on your study and fail to meet the terms of the surety bond, next time your rates are going to be higher. If you hire X person as a reviewer, that will affect your rates, too, if they have a track record of overseeing papers that end up with retractions. If you are “uninsurable” in this way, that’s a big red flag for anyone cutting a funding cheque.
Where’s the trace-ability, now? A member of the taxpaying public has no idea how much money is being tossed at bad science, and this offers a means of transparency to the whole process.
Leonid Schneider is very active on PubPeer, but his views expressed in a recent PubPeer case deserve special attention, in paticular his defense of his statement “I think I owe an explanation to certain Unreg(s) why I think I know what I am taking about when I speak about science integrity”:
https://pubpeer.com/publications/209CA2DF493322A5B5470F3B8EEDA0#fb37749
I wonder if articles containing faulty results will no longer be retracted after implementation of measured advocated in the post. Editors of the journals too are usually active members of the field and there are certain ways to influence their decision to issue a retraction statement.