Paper co-authored by sleuth Elisabeth Bik marked with expression of concern

A paper with scientific sleuth Elisabeth Bik as a co-author now has an expression of concern. It dated back to her time at the now-defunct startup uBiome and described research that the company used to develop a clinical test of bacteria living in the human gut – and that she raised concerns about some years ago.

The article, “16S rRNA gene sequencing and healthy reference ranges for 28 clinically relevant microbial taxa from the human gut microbiome,” was published in PLOS ONE in 2017 and has been cited 39 times, according to Clarivate’s Web of Science. 

The expression of concern detailed the journal’s investigation into allegations that some of the samples in the paper weren’t suitable for determining a healthy baseline of the human gut microbiome — being from infants, people who might have recently taken antibiotics, and pets — and the authors’ responses. 

It’s a long notice, but this paragraph sums up the concerns: 

The PLOS ONE Editors issue this Expression of Concern to inform readers of the concerns about a possible non-human origin of one of the samples included in the study, and about the limitations of the study design with regard to drawing conclusions about clinically relevant levels of taxa  and that routes to request access to restricted data do not appear to be functional at the time of preparation of this notice. 

As a prolific scientific sleuth, Bik has identified duplicated images in hundreds of papers in PLOS ONE and notified the journal about them, resulting in a slew of retractions and other editorial actions. She’s commented in the past that the journal has not yet acted on a lot of what she’s sent them from as far back as 2014. 

Bik was one of the scientists who initially raised concerns about uBiome’s research that was reported in the paper, according to a 2019 report in Business Insider. She and others told the website that the dataset in the paper likely contained samples from infants and pets, which could skew results on the clinical test the company was developing, later sold as SmartGut. 

When Bik shared her concerns with uBiome executives, they waved her off, she told Business Insider, and the paper had already been submitted to PLOS ONE. 

The Business Insider report may have spurred the journal’s investigation of the paper. A journal spokesperson told the website “that it was not aware of the concerns with the paper and would look into them,” according to the article.  

In the expression of concern, the authors provided more information about the study methods, an updated email address for the corresponding author, and contact information for the trustee of the company’s bankruptcy estate with responsibility for retaining study data. They also pushed back on the allegation that the dataset included an animal sample: 

Following publication of this article [1], concerns were raised about the origin of stool samples used to generate the microbiome dataset. Specifically, concerns were raised about the possible inclusion of samples from participants affected by health conditions or antibiotic usage, and non-humans.

The PLOS ONE Editors followed up with the authors, and it was confirmed that fecal samples were collected by participants, and those included in the study reported being healthy by survey; the study design did not exclude participants based on age. The reported age range for participants is 19 days – 103 years. 

The first author has stated that an audit of customer service data held at uBiome in August 2019 identified information indicating that one of the samples submitted by a uBiome user, and labelled as a human gut sample, may have had a non-human origin (sample 757 as listed in S3 Table of [1]). This information was stored on a customer support system separate from the anonymized database used in the study and was unknown to the authors at the time of the study. 

The first author provided the following additional information: 

Evaluation of the data from the suspected non-human sample 757 indicates that this sample is not an outlier for the abundance for any microorganism. Recalculation of the values for the healthy ranges with sample 757 removed from the data set generates the same numerical values for the upper limit of the healthy range as those indicated by the red lines in Fig 3 of [1], previously corrected in [2]. Thus, because of the large sample size of this study, and the normal distribution of microorganism abundances in sample 757, inclusion or exclusion of sample 757 from the calculation of the healthy ranges does not alter the results and conclusions presented in the published article [1].

The corresponding author clarified that the uBiome user who submitted sample 757 had more than one sample in their account, which introduces the possibility that the non-human sample was not included in the study. They note that sample 757 clusters with the rest of the human samples in the study, which they say suggests a human origin for this sample.

The expression of concern further questioned whether data from the self-selected participants – people who volunteered to send in poop samples – could really be a reliable standard for a clinical test: 

The PLOS ONE Editors carried out an assessment of the published study in consultation with a member of the Editorial Board who advised that while the study provides data for the relative abundances of microbial taxa in the selected healthy cohort by 16S sequencing, it does not provide a basis to define clinically relevant relative levels of target microbial taxa and provides minimal data on the use of 16S sequencing for clinical diagnosis. It was additionally noted that the healthy range results reported from a self-selected sample of individuals interested in obtaining gut microbiome data may not be generalizable to other populations. 

In comments to Retraction Watch, Bik said she thought the expression of concern was out of proportion with the paper’s issues: 

The paper in question describes a citizen-science project, in which people were invited to sample their stool samples using the uBiome Explorer kit, and answer questions about their health through a questionnaire. A set of 897 stool samples from different participants self-reporting to be healthy were selected, and the relative abundances of different, clinically-relevant, microbial groups were determined through 16S rRNA sequencing. The purpose was to determine what the abundance ranges were for these microbes in healthy persons. 

The Expression of Concern mentions two potential problems with the paper. 

1. After the paper came out it came to light that “one of the samples submitted by a uBiome user and labelled as a human gut sample may have had a non-human origin“. 

According to the PLOS ONE evaluation, it is not even certain that it *was* a non-human sample, it is just a possibility.  Since this particular sample was not an outlier among the data from the other 896 human stool samples, it might actually have been a human stool sample – pets or kitchen floors usually have different microbiomes than human stool samples. So even if one of the participants swapped their cat or floor sample instead of providing the requested human stool sample, such a sample would probably have had a different composition than a human sample, and would have looked like an outlier. But it didn’t. In addition, it is also just one sample out of 897, so 0.1% of the samples. Removing this particular sample did not have an influence on the healthy ranges determined in this paper. Issuing an Expression of Concern for a study in which one of nearly 900 samples might or might not have been incorrect seems a bit harsh. 

2. The other potential issue with the paper is that results reported from a self-selected sample of individuals interested in obtaining gut microbiome data may not be generalizable to other populations.

The argument that this paper’s cohort is not generalizable to other populations is true for many studies done on the population of a certain region or country. Any study done on a particular cohort might not be applicable for other cohorts. MIcrobiome studies done on stool samples from subjects living in the US will give a different outcome than those done on e.g. the Hadza people in Tanzania. A study on the health outcomes during pregnancy done at Stanford University might give a very different outcome than similar studies done at Wayne State University in Detroit or a small community hospital in Mozambique.  But that does not necessarily mean that there is no value in a study done with one particular set of individuals. Although I agree that this study might be limited in terms of applicability to other populations, this is true for many, many other studies focusing on one set of subjects, and it is unusual that this would lead to an expression of concern. 

Expressions of Concerns are usually applied to papers where there are serious concerns about a large part of the data, not when there are doubts about one sample in a set of nearly 900. In the past, PLOS ONE has issued EoCs for papers with multiple overlapping or potentially photoshopped images such as DOI: 10.1371/journal.pone.0052408 or 10.1371/journal.pone.0114021 . Or, like in 10.1371/journal.pone.0231520, where the wrong cell line or a contamination might have affected the majority of the experiments, or 10.1371/journal.pone.0060289, where primate research might have been conducted without the proper ethical permits. 

I am not sure if the concerns in our paper rise to the same level as the examples listed above. I feel this issue might be better addressed with a correction.

Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].