Peer reviewers are supposed to be experts in their fields, competent enough at least to spot methodological errors, wayward conclusions and implausible findings. But checking references? Apparently, not so much.
A journal about academic medicine has retracted a 2020 article because its reviewers and editors didn’t bother to confirm that the references said what the authors said they did — and because when pressed, the corresponding author couldn’t provide the underlying data for the paper.
The paper, “Medical students’ perception of their education and training to cope with future market trends,” appeared in March in Advances in Medical Education and Practice, a Dove Press title. The author was Mohamed Abdelrahman Mohamed Iesa, a physiologist at Umm Al-Qura University in Saudi Arabia.
The article presented the results of a survey of 500 medical students at 10 schools in the United Kingdom. It purported to find that:
there is a clear need for courses on professionalism and management among medical students and that institutes need to keep up with these emerging needs in terms of training.
Evidently those medical students who become academics also need some instruction on how to review a paper so they can do a better job than what happened in this case.
Shortly after publication, the journal learned that some of the references in the article did not support the conclusions. As the retraction notice explains:
We were notified of potential discrepancies between three articles cited in the Background Studies section of the published article. The cited articles do not appear related to the data they were being used to describe. The studies cited were listed as references 19, 20 and 21:
● Lunt N, Horsfall D, Smith R, Exworthy M, Hanefeld J, Mannion R. Market size, market share and market strategy: three myths of medical tourism. Policy Politics. 2014;42(4):597–614. https://doi.org/10.1332/030557312X655918
● Lempp H, Seale C. The hidden curriculum in undergraduate medical education: qualitative study of medical students’ perceptions of teaching. BMJ. 2004;329(7469):770–773. https://doi.org/10.1136/bmj.329.7469.770
● Könings KD, Brand-Gruwel S, Van Merriënboer JJ. Towards more powerful learning environments through combining the perspectives of designers, teachers, and students. Br J Educ Psychol. 2005;75-( 4 ) : 6 4 5 – 6 6 0 . h t t p s : / / d o i . o r g / 1 0 . 1 3 4 8 /000709905X43616
The data being referred to by these references appears to have come from another study which was not cited:
● Rouhani MJ, Burleigh EJ, Hobbis C, Dunford C, Osman NI, Gan C, Gibbons NB, Ahmed HU, Miah S. UK medical students’ perceptions, attitudes, and interest toward medical leadership and clinician managers. Adv Med Educ Pract. 2018;9:119–124. https://doi.org/10.2147/AMEP.S151436
The notice states that Iesa was “unable to provide a satisfactory explanation for the alleged discrepancies.” But there was another problem:
The author was also unable to provide satisfactory raw data for their study and could not provide details of the questionnaires used for the survey or details of the UK medical schools or students described in their study. The editor determined the findings of the study were no longer valid and requested for the article to be retracted.
We emailed Iesa but he did not respond. [Update: He responded in the comments.] We also emailed the editor who accepted the paper, and received this reply from Sabina Alam, the director of publishing ethics and integrity for Dove:
This journal has a single-blind peer review policy, with this paper reviewed by two reviewers prior to publication. The referencing issues weren’t picked up through peer review but, having worked through this case, this is an isolated incident. Once we were alerted to the potential issue, our investigation procedure ensured that wider issues around discrepancies in the data were also picked up, which collectively led to the decision to retract the article rather than issue a correction. We don’t envisage an immediate need to change our policies, but continue to work across our journals to ensure best practice guidance is provided to reviewers and academic journal editorial teams, to ensure high standards of editorial decision-making.
Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].
If you do a mass of peer reviewing the checking of each single reference is far too onerous, especially in systematic reviews where there are often 100+. I do check references if they make a claim that I consider odd, especially when they claim one of mine says something quite different from what I remember. I also check data references where the data look odd. It is remarkable how often authors in systematic review either extract data incorrectly, or the extract the wrong data, or mix us SD and SEM, or fail to read papers correctly.
I will also often run my own searches for any other SR or MA, or RCTs.
But it is time-consuming, and I’m doing it for free. So there will be limits.
Echoing Andrew Moore’s comments about reviewing as time-consuming work done for free and the impossibility of checking every reference, mainly those that made odd claims…. I recently reviewed a paper in which the authors apparently counted on reviewers not checking references that looked “probably OK”. They cited a paper I was familiar with and I didn’t remember the particular feature they cited; that led me to find neither that paper nor 10 others were appropriate references, which unraveled the authors’ “emperor’s clothing” attempt to say that their results somehow supported their hypothesis, and showed that their conclusions were balderdash. As a reviewer without a vast amount of experience, it was eye-opening for me.
My advisor asked me to peer review a paper for him at the start of the pandemic. He gave me one from a group he knew did good work so he didn’t expect there to be many issues which was nice for me. It was a pretty short paper, but yeah there was no way I was going to look through all 40+ references. I spot checked a couple, and saved a couple other ones that I wanted to read, but otherwise checking references would be a full time job. I don’t even want to think about the time it would take on a full review paper.
Of course then there’s the issue that many papers don’t have the best references. Either the reference is to the wrong paper, or it’s clear the authors misread something. I don’t fault people for that, they are very human mistakes, but this is one of the biggest weaknesses of science IMO.
Study below estimated major and minor quotation error rates of 11.9% (95% CI 8.4-16.6) and 11.5% (95% CI 8.3-15.7) . So its a very common problem . Agree when peer reviewing you can hope to spot errors but there are issues like selective citation that its very random if you will find the problems. I have reviewed manuscripts where I’ve pointed out these kind of errors and editors haven’t insisted they be corrected.
Quotation accuracy in medical journal articles – a systematic review and meta-analysis.
PeerJ. 2015; 3: e1364.
Published online 2015 Oct 27. doi: 10.7717/peerj.1364
PMCID: PMC4627914
PMID: 26528420
Quotation accuracy in medical journal articles—a systematic review and meta-analysis
Hannah Jergas and Christopher Baethge
Thank you for what you do at Retraction Watch.
Unfortunately as an editor for one journal, a review editor for several others and reviewer for a number of other journals, I have seen this a number of time times and have heard from other editors that they have also encountered this problem as well. I am sympathetic with the commentators above saying it is very difficult and tedious as reviewers to check every reference particularly in fields like biomedicine where it is not unusual to reference 30 or 40 articles in typical scientific paper and even more in a review paper. It is sad busy scientists who are willing to donate their time to reviewing even have to consider checking references. They should focusing on the science but unfortunately we need to at least spot check references because this is a real problem.
In fields where where large numbers of references are expected, there are authors who cut corners and apparently copy reference lists from other similar articles referencing articles perhaps based on their titles without even reading them. On occasion, when I have checked references, comparing them to what the authors have attributed to the reference, there is no relationship. This is blatant scientific misconduct and should be treated as such. These manuscripts need to be rejected, the authors banned from publishing in the journal and reported to their institutions for committing scientific misconduct.
Interesting article.
I can only concur with the others that it is not feasible for reviewers to check every reference. And it is not the reviewers’ responsibility that every aspect of the paper is correct and stands to scrutiny.
As much as I respect this website, it’s a bit offensive to say that “reviewers and editors didn’t bother to confirm that the references said what the authors said they did”.
This would be a nice goal, but the change needs to come from elsewhere if we are going to reach anything like it. And that’s not going to happen during my lifetime.
Yes, Retraction Watch, do you check every citation of the papers you expose? I think you do not.
I do a fair amount of reviewing and often find problems with the references, including incorrect author names and citations. I do check the content if I am not familiar with the cited article or if something looks off. Not all articles cited as references, however, are easily obtained, if at all, by reviewers. Some of them, for example, might be older articles that are not always easily accessed. And what happens if an article gets into print that misrepresents the content of some references? It seems unlikely that there would ever even be a correction or a retraction.
This is a great problem in my view. Unfortunately it is not episodic. In my experience this is a common and diffuse problem involving even papers of strong groups of research. I frequently need to trace back some references of highly quoted articles and I find that they are not appropriate. Then, I remain baffled as I cannot find any support for important statements usually, located in the introduction, which makes the basis for building up the reasoning of the paper.
This is not fiddlesticks. This is a very important issue. How to deal with this? Well, first of all I believe that the current push to publish quickly is wrong. When you ask reviewers to make their job in two weeks and you peste them with emails remembering the due date, it is as saying “doesn’t matter how you make your review, it matters that you do it quickly”. Checking also references, therefore, becomes an illusion.
When I was young, mean time for receiving an answer from the editor was 3 months! At that time you had the time also to check references if you wanted to do it.
Citing papers that one has not read without properly acknowledging that one only knows it from second-hand information is scientific misconduct, right? Articles being difficult to obtain does not seem a good reason to break that rule, imho.
In the last century, I was as asked to review articles in a very specialized field. Being aware of many of the relevant articles, I always checked citations. In doing this, I developed a number of rules.
First, if there were too many (greater than 50%) non-primary citations, the article was likely to be rejected or rewritten. Non-primary citations were things like review articles, which were just barely acceptable; or articles citing review article, not acceptable; or, being lazy and using another author’s interpretation of an article; or, opinions found on web sites which cited non-primary sources exclusively. This latter almost guaranteed rejection.
Why were review articles not acceptable? These articles always had the primary citations and if the author could’n be bothered to verify the information/conclusion, then what else did the author miss? Also, as an researcher in the field, I often found mis-interpretations of my articles. While I couldn’t stop this, I could try and discourage the spread of these errors. Failed miserably.
Second, I would spot check about 25% of the citations, concentrating on web citations. If I couldn’t find more than two, the paper would be recommended for correction, if there were no other problems. Greater than two, it was the rejection pile.
This practice led to the following response from a lead author: “We find this comment unacceptable. If we knew that the reviewer was going to actually check the citations, we would have been more careful.” Vindication.
I’ve never heard of anyone being banned from a journal in my field. The refusal of major societies to take this seriously (hello, OSA!) is one of the reasons I don’t review papers anymore. I got sick of shitty plagiarists submitting the same paper over and over.
I think Web of Science introduces a problem in this area.
When you do a research topic search, Web of Science will pull articles in the search results that have your topic in the title, abstract or author keywords. BUT also will use KeyWords Plus as a source – “the data in KeyWords Plus are words or phrases that frequently appear in the titles of an article’s references, but do not appear in the title of the article itself.”
So this means that you fetch articles that are not about a topic but referred to an article about the topic.
Now if you don’t actually read these articles and just use them to pad your reference list then you will have irrelevant articles.
So many red flags here. Single author not based in UK. Somehow gets access to phone numbers of students from 10 UK medical schools to send them a survey through Whatsapp. No mention of ethics review.
Shoddy “science”.
tbh, many reviews are so parsimonious and non-specific that not checking references is the very least of the review deficiencies. It’s often a huge struggle to get any engaged and qualified reviewers.
I have rejected many, many manuscripts on this issue and called for editors to issue a formal complaint to the authors’ affiliation. It is a classic ploy. Offenders think that, just because they’ve cited the key papers (almost in passing during the introduction in a group of papers, along with a deliberate misrepresentation of findings) reviewers won’t notice that their experiment and conclusions are identical. Very common, I’m sorry to say, in submissions from China in my field.
Dears Retraction Watch Team,,
Thank you for bringing this issue here. I’m the author of this manuscript. I didn’t received your email regarding this issue. could you please resend this email to me.
Actually this manuscript is totally questionnaire based and participants never asked to mention their institutes to avoid bias.
I have cited those three references in the Introduction part of my manuscript not in the result or discussion and I think this paper reflect my perception in this field and no one of those authors have any objection about citation of those three papers.
Please note the following points regarding citation of those three references according to my perception and philosophy as a medical educationist:
Market size, market share and market strategy: three myths of medical tourism
I have chosen to cite the above reference as matches my submitted article with respect to the following points:
1. The reference takes into consideration the effect of management skill for the future medical practitioners and the thereafter effect on the medical touristic market.
2. Scan of the medical tourists, a terminology used in both the articles (the cited and mine)
3. The cited article used QUALITATIVE STUDY, which is also used in my article.
4. The cited article directly correlated to the management skill and its effect on market and the same was considered in my article.
The hidden curriculum in undergraduate medical education: qualitative study of medical students’ perceptions of teaching:
I have chosen to cite this article as it also related to my accepted article with regard to the following points:
1. The study linked between the changes related to the management skill gain and the perception of the teachers of what their medical students might require of management skills. These were, as well, interrelated with the teachers’ loyalty to specific medical specialization.
2. The study relates the competitive environment and bulling during clinical studies and relation of these to the students’ knowledge gains.
3. The study also linked between effects of the hidden curriculum with respect to the acquiring of the professional skills which represents a focal point for my study.
Towards more powerful learning environments through combining the perspectives of designers, teachers, and students
I have chosen to cite this article as it correlate to my article with the respect to the following point:
Both articles reflect on the effect of the learning environment where future medical practitioners will acquire the needed medical professionalism. Based on this my study also emphasize on the need for a changing this environment in such a way to enhance these students quality.
UK medical students’ perceptions, attitudes, and interest toward medical leadership and clinician managers
With respect to the reason why didn’t cite the above reference. I read this article solely after your email and was, unfortunately, never come my notice before. I found it interesting with respect to the students’ knowledge gain, as this part of medical education grows very rapidly, and I feel it could be of importance to cite this article as well. Though of the quality of this reference, it differs from my article in the following points:
1. My article directly relates between the expected deficiency in the students’ professionalism for a reason related to curricula that never covered and not to the teachers’ behavior as the case in Rouhani et al article.
2. My article related to the six types of professionalism and the effect of each on the students’ acquired skills, whereas Rouhani et al article took the effect from the acquired training of the students in the surgery department point of view.
Dr. Mohamed A. IESA