Analysis casts doubt on bone researcher’s body of work

A new analysis of more than 30 clinical trials co-authored by a bone researcher based in Japan is casting doubt on the legitimacy of the findings.

Yoshihiro Sato, based at Mitate Hospital, has already retracted 12 papers, for reasons ranging from data problems, to including co-authors without their consent, to self-plagiarism. Most of these retracted papers are included in the analysis in the journal Neurology, which concluded that Sato’s 33 randomized clinical trials exhibited patterns that suggest systematic problems with the results.

Other researchers have used similar approaches to analyze a researcher’s body of work — notably, when John Carlisle applied statistical tools to uncover problems in the research of notorious fraudster Yoshitaka Fujii, and Uri Simonsohn, who sniffed out problems with the work of social psychologist  Dirk Smeesters.

Author Mark Bolland of the University of Auckland told us he was surprised by his findings:

…we were very surprised by the strength and consistency of the different statistical analyses, which showed that the baseline data for the randomised treatment groups presented in the papers were strikingly similar, and not consistent with the treatment groups being formed by chance (ie that randomisation took place). The productivity of the small research group (33 trials over 15 years), and their recruitment and retention rates of frail elderly patients with high rates of comorbidities seemed fairly incredible. Also, the consistently large reductions in hip fracture rates from many different agents in their trials seemed implausible and was inconsistent with results from trials for the same agents conducted by other groups.

We contacted Sato, who responded to many of the paper’s allegations. Regarding his ability to recruit so many patients in such a short time, Sato told Retraction Watch:

We recruited a large number of neurologic patients during 15 years. The lead author (Y. Sato) took charge of patient follow-up with support by 2 additional physicians of Mitate Hospital. Mitate Hospital has 410 beds and is located in the same Chikuho Province…only five professional neurologists practiced in two hospitals―Mitate Hospital and another one―and physicians who could diagnose and treat elderly neurologic  patients were very few in view of the population in Chikuho Province. For example, our hospital admits mild stroke patients without any impaired consciousness and patients with unconsciousness were transferred to other hospitals as we described elsewhere (Neurology 2005; 65:1513-4). Thus, many stroke patients in our hospital met the inclusion criteria of the studies.

In some studies on stroke and Alzheimer’s disease (AD), we included patients from two collaborating hospitals and a nursing home in addition to the patients treated in Mitate Hospital. The nursing home is affiliated with Mitate Hospital and most of the institutionalized patients were treated as out-patients of the hospital.

Sato debated the statistics underlying the conclusions about randomization, and argued that the patient population he examined may be predisposed to responding better to treatment than others:

The remarkable outcome of the effects of interventions ―administration of vitamin D and bone resorption inhibitors (A22) ― on fracture or bone density may be attributed to the fact that our study subjects were neurological patients, who may have higher risk of fracture without any intervention. Generally patients with Parkinson disease have higher levels of impaired movement as compared to postmenopausal women. This fact may explain the occurrence, in these patients, of immobilization-induced hypercalcemia due to enhanced bone resorption, which, in turn, may inhibit PTH secretion resulting in the suppression of vitamin D activation. These characteristics of bone and calcium metabolism in patients with Parkinson disease may have predisposed them to high effectiveness of the interventions.

You can read Sato’s entire response to the Neurology paper here.

Bolland cautioned that he and his co-authors applied statistical analyses to the group of studies, so the results can’t specify whether one paper is more reliable than the other. Nevertheless, in 2013, they began notifying individual journals of their findings, Bolland told Retraction Watch:

To our knowledge, all but one of the retractions to date have occurred either directly or indirectly as a consequence of our analyses, since they followed the receipt by the journals concerned of our manuscript.

The project that culminated in the paper “Systematic review and statistical analysis of the integrity of 33 randomized controlled trials” began in 2012, after a conversation with a colleague flagged Sato’s work, Bolland added.

[The colleague] mentioned that she had come across three trials with identical results while doing various systematic reviews. She asked what we knew about the research group of Dr Sato, who published trials frequently in the field of osteoporosis, which our research group is active in…Even a superficial viewing of several of the trials in 2012 raised a number of questions, so we decided to investigate further by systematically reviewing all the published trials of this group.

Bolland and his colleagues’ approach to analyzing a researcher’s body of work reminds us of that taken by John Carlisle, an anesthesiologist in the United Kingdom, who used statistical tools to determine whether Yoshitaka Fujii’s work was legitimate — and triggered the retraction of a record-breaking number of papers (183). Carlisle told us he believes the Neurology paper is a “great piece of work:”

The weight of evidence makes it likely that all of Sato’s papers should (?will) be retracted…

Carlisle added that he’d already analyzed some of Sato’s clinical trials as part of a larger survey of more than 500 such trials published in JAMA, and performed a quick analysis of the papers listed in the Neurology paper. He agreed with the authors that the baseline data seemed improbable — in one paper, he calculated that the odds an experiment would have such an extreme distribution of continuous baseline data were 1 in nearly 2.3 million.

Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.