I was once required to testify in a court case. My lawyer gave me a few pieces of advice, but he repeated one several times, which may be why I remember it. “Never say never,” he said. Or, conversely, never say always. Declarations of absolutes present opposing attorneys too wide an opening. They need to identify only a single example to contradict. In trial courts, one cannot get away with making reckless absolutist claims unchallenged.
In academic scholarship, however, it happens all the time.
Meet the dismissive literature review, in which an author at the beginning of a journal article declares the published research literature on the topic either nonexistent or so poor in quality that all of it is … dismissible. Typically, no evidence supports the claim. You’ve seen the claims yourself (e.g., “little previous research has, …” “few studies have looked at …,” “there is no research on …,” etc.). With one type of dismissive review — a firstness claim — authors boldly declare themselves to be the first in the history of the world to study a particular topic (as in, “this is the first study of …”).
In academia, declarations of a void in the research literature are rarely challenged. As long as a few unknowing, uncaring, or otherwise cooperative reviewers and editors let the statement slide, it passes unimpeded into the world of scholarship and becomes what I call a dismissive literature review. No one with a self or public interest in countering the claim is offered an opportunity to challenge.
The size of my collection of such reviews surpassed 1,000 some time ago, despite its limitation to a single, relatively small topic — U.S. education policy — and a small proportion of researchers in that field sometimes labeled “celebrity scholars.” Those are researchers blessed with public relations offices and information dissemination budgets supporting the promotion of their brand (think government-funded research centers, think tanks, the most prestigious universities).
Serendipity started the process. I encountered most of the collected items in the normal course of my professional reading. I first took note of false dismissive reviews on topics I had deeply researched, topics which I knew for a fact contained robust research literatures. After a while, I started to notice dismissive literature reviews throughout scholarly research.
When I am deeply familiar with the dismissed literature, I have added notes to the reviews, typically containing the names and publication years of some of the dismissed research. Granted, many reviews remain in the Dismissive List with notes not yet added. Conducting compensatory reviews for over 1,000 separate claims poses a considerable time commitment, as honest, thorough literature searches can take days, weeks, months, or years, depending on the topic, the number of searchers involved, and the resources at their disposal.
Of course, in dealing with so many separate details, I may have made mistakes. With only a single exception, however, I can vouch for one reliable characteristic of the Dismissive List: the author in each case provided no evidence to support his or her claim of a research literature’s nonexistence or alleged inferiority. In each case, no mention was made of where, how—or even if—the scholar searched for previous studies, as is routinely provided in meta-analyses. The dismissive claims are not research results; they are unsupported declarations.
The Dismissive List comprises dozens of worksheets, organized by individual , group of scholars or journalists. Journalists include themselves in this mode of information suppression when they repeat a scholar’s dismissive review, without any effort made to verify the claim.
Each entry in the Dismissive List includes fields for:
- Author (s)
- Co-author(s)
- Dismissive quote
- Dismissal type (simple dismissal, firstness claim, or denigration)
- Title of article or document
- Source location (e.g., publication name)
- Hyperlink to article
- Funders of study, when known
- Notes on previous, dismissed research (authors, dates)
- Notes on other misleading claims (when applicable)
For the most part, the list includes statements made by “serial dismissers,” scholars who dismiss repeatedly on a variety of topics. This is done to help counter the argument that they might be innocent, did try to look for previous research, and simply could not find it. In some cases, they dismiss a research literature that is hundreds or thousands of studies deep. And, when they do that repeatedly across a variety of topics, the odds their dismissive behavior could be innocent fade to miniscule.
Some articles lead with a dismissive review. “This is the first study of X” may be the first sentence in the abstract and the introduction (and perhaps also the conclusion). The author crowns a research article with a statement unsupported by any evidence.
Some scholars seem addicted. They place dismissive reviews in the article abstract, introduction, literature review section, conclusion, discussion, and perhaps sprinkled elsewhere throughout. Some individual articles contain a dozen or more. One might assume a single dismissive review near the front of an article would be sufficient for driving home the point – namely, an alleged absence of previous research. Why repeat the same message throughout from beginning to end?
I suspect the behavior is not intended to get that point across but, rather, another.
A dismissive review implies one is familiar with an entire research literature …that one is a topical expert. The desire to exploit any possible opportunity to proclaim expertise may motivate the repetition. (Indeed, if no previous research on the topic truly existed, the dismissive scholar would be the world’s foremost expert after completing a single study.)
Unfortunately, someone falsely claiming to know an entire research literature is the opposite of an expert. And every occasion of a policymaker or journalist seeking their false expertise is an opportunity lost, when they could have consulted a genuine expert.
Claiming an absence of research on a topic comprises the entire substance of some articles, without any evidence of said absence, or of having looked for it. Apparently, some editors see no harm in publishing an article wholly devoted to declaring a void (without evidence) and a need for more research. Research is what we do, and more research is always good, isn’t it?
Some highly successful scholars have employed dismissive reviews throughout their careers. Which raises the question: Have they profited more from the quality of their work, or the substantial advantages provided by dismissive reviewing?
Those benefits include time savings; avoiding contradictory evidence and debate; adding to one’s citation totals, while not adding to the citation counts of their rivals; and establishing (false) bona fides as an expert and research pioneer, which increases likelihood of press coverage and grant funding (to “fill knowledge gaps”).
Indeed, two of the most celebrated and rewarded scholars in the Dismissive List have accumulated two of the longest dismissive review worksheets. Each has been publishing for over 30 years and written hundreds of dismissive literature reviews throughout their careers. Their behavior serves as a model to their graduate students and others who might aspire to their level of prestige and professional attainment.
In the case of one scholar with almost 300 dismissive entries, most of the literature reviews in the list are rebutted with an extensive compilation of previous studies . For one line of research, for example, the author claims a lack of studies and promotes one of his from over 30 years ago, conducted at a location he will not identify (confidentiality!), using outcome measures he will not identify, and producing a data set he will not provide (again, confidentiality!).
In fact, other studies on the topic date as far back as 1927. At least 79 relevant studies were published between 1927 and 2019. Meta-analyses or research summaries were published in 1981, 1983, 1983, 1983, 1984, 1990, 1993, 2005, 2005, 2006, and 2017.
Typically, the only research this author acknowledges on any given topic is that conducted by colleagues in his small, mutual support circle. If only a handful of studies conducted by him and his close colleagues support his views on a topic, they become the only studies that exist on that topic.
How do they get away with this? One way is by hedging the dismissive declaration. One doesn’t declare that no previous research exists, just that the amount is “little,” “thin,” “scarce,” “scanty,” or “few” and its quality not “rigorous,” “new,” “solid,” “systematic,” or “robust.” As the dismissed research is not cited and the dismissed scholars are not identified, readers are given no assistance in finding and reading that work and deciding for themselves.
Hedges can vary dramatically in meaning. “There have been no rigorous studies of X” could mean no previous studies have employed the analytic method one prefers (e.g., multiple regressions), or it could signify a lack of studies by others in one’s field (e.g., other economists) or those by anyone one considers important.
Another way celebrity researchers get away with dismissive reviews: money.
A majority of the reviews on the Dismissive List were funded by the federal government or wealthy foundations. That success also suggests that the scholars receiving the funding may have used dismissive reviews in their grant applications as motivation for the funding, as in: we need to know X, there is no research on X, so give us money to study X. (To investigate this hypothesis, I once asked a few research-funding foundations for copies of the original grant applications for some of their funded projects. Each declined.)
When those dismissive reviews are false, funders ostensibly dedicated to increasing knowledge may instead be reducing it.
Loath to mimic behavior I criticize, I contacted some of the researchers featured in the Dismissive List and offered them an opportunity to comment on this article. I chose the 14 individuals who contributed the most dismissive reviews to the list — from at least 40 up to just shy of 300.
Only one replied. According to Eva L. Baker, the longtime co-director of the Center for Research on Educational Standards and Student Testing (CRESST), who has 45 entries on the List: “We could argue about the state of evidence on any of a number of topics.”
All the above may make one wonder. How many of the most ambitious scholars conduct honest research literature reviews and thoroughly report those findings in their journal articles? With little enforcement or accountability, the benefit of dismissing a research literature without bothering to look for it far outweighs the risk of getting caught in the lie.
For many scholars, the default literature search has become, “Let’s not and say we did.”
Richard P Phelps is the author of The Malfunction of US Education Policy: Elite Misinformation, Disinformation, and Selfishness. Read his other Retraction Watch guest posts here.
Like Retraction Watch? You can make a tax-deductible contribution to support our work, subscribe to our free daily digest or paid weekly update, follow us on Twitter, like us on Facebook, or add us to your RSS reader. If you find a retraction that’s not in The Retraction Watch Database, you can let us know here. For comments or feedback, email us at [email protected].
Great read.
Some small typos: celebrity schlars, Granted, t…
Fixed — thanks.
While I can completely understand the author’s point, I would like to add another point of view: that is how we educate our scholars!
Every grad student and every researcher is taught to do a literature review and use this review as the main argument for their research. The argument is always the same. There is a lack of research on this small, singular topic, so I researched what I did. I would go even further and argue that this is a direct result of Positivism as the dominant philosophy of science.
The negative results that follow are ignoring the research beyond one’s bubble and rejecting other people’s research in peer review.
That’s not how I taught the students. It’s probably difficult for a graduate student to find a unexplored area or question to research. I told them you should critically evaluate the literatures, which might have this or that flaws, samples, measure, procedure, data analysis etc . These can be the gap or problem statement too
Possible typo: “celebrity schlars.”
Fixed — see HB comment and reply above. Thanks.
How would you say the absence or insufficiency of previous research on a topic should be evidenced? I can only think of stating that a certain method of literature search was used, but this seems to me to be weak evidence.
Other ideas?
I think I have made such dismissive reviews, not in press (I think) but at times in non reviewed papers. And I have thought the same to myself, how does on demonstrate a lack of research on a topic?
Your suggestion is a reasonable one. You believe it is weak because you’re only thinking of the direct evidence it may contain. Consider that a detailed description of a search method can be examined, and even replicated by others. Several things could be established from that, whether it was reasonable for the author to have conducted such a search, whether they drew a reasonable conclusion from it, and paths for future researchers to explore.
I had the opportunity once, because of a dismissive claim I made in a submitted paper, to be informed how wrong I was by colleagues. They gave me lists of papers they felt that I had merely replicated. I had the opportunity to go through those lists and either correct my opinion or take the list of papers from experts in the field and use their belief in what those papers demonstrated to support my claim. I was grateful for that opportunity. I regret that I never took it and focused more on their negative disposition.
The flip side of this are the often nebulous novelty requirements enforced by journal editors. I believe these dismissive phrases are a lazy or clumsy attempt to signal novelty. The publication system actually incentivises this. But thank you for this essay. I shall uses it as a reference when (not if) I see dismissive phrases without evidence as a reviewer.
Historians of science (I am one) spend considerable time contesting and refuting spurious claims to novelty. Moreover, in certain areas of research, we create a “stemma” through the published literature, following misspellings and other errors in journal title abbreviations through sequences of papers, which we interpret as evidence that the paper was not actually consulted by the author but copied, unread, from a previous paper. This technique we learned from classical scholars who trace the history of manuscripts through the (innocent) repetition of scribal errors. In the more modern case of literature reviews, a good deal more than innocent error involved – adding pretension of thoroughness in review (via the creation of a fictional record of papers “consulted”) to the list of relevant papers entirely ignored, willfully or carelessly.
I am interested in knowing more about this. Can you direct me and us to examples of this “stemma” technique that historians of science use to evaluate novelty claims? It sounds somewhat like what psychometricians use to detect classroom test cheating (essentially, looking for high correlations of common wrong answers). Historians of science may be society’s most essential backstops in our search for truth.
I agree with Richard Phelps’s discussion of dismissive reviews. However, I have purposefully, knowingly made a dismissive statement–for dramatic effect–in three successive publications regarding parental alienation (PA) misinformation. So far, no one has challenged this dismissive statement: “No PA scholar–since the seminal description of PAS by Gardner (1985)–has stated that all children who manifest contact refusal were indoctrinated to fear the rejected parent by the favored parent.” (Bernet and Xu, Behavioral Sciences & the Law, 41, 231-245) I reviewed about 400 publications regarding PA theory before making that statement. If anyone ever locates an exception to my dismissive statement, that will be a great topic for further discussion.
I don’t follow this essay. This so-called dismissive lit review is very good and necessary; it should definitely be done to show the novel aspects of the research (if any, or lack of them). I do this in all my papers. As a reviewer and editor, I always ask the authors to clearly state what aspects of their study is completely or partially new and how they can be of interest.
It is a necessary step to show the readers and the reviewers why the study was conducted in the first place and why it may be of interest. It is an important part of something called “study rationale”!
Even CONSORT / STROBE / etc. checklists require authors to clarify the study rationale by reviewing the literature: The authors should clarify the abundance or lack of previous studies as well as any controversies in the literature that would benefit from more studies. This is ‘science writing 101’.
ps. It is OK (not preferable but OK) if a study is just confirmatory without any novelty, especially if the literature is controversial. In any case, with or without novelty, the authors should disclose the novelty or lack of it.
You’re missing the point. This essay is not about researchers claiming novelty; it’s about researchers conducting poor literature reviews and outright lying about novelty. In my opinion, this should count as scientific malpractice.
I also strongly disagree with your attitude that it is “not preferable” for a study to be “merely confirmatory without any novelty.” By this logic, nobody is incentivized to do replication studies. If anything, the current state of certain disciplines and even the mere existence of Retraction Watch itself is compelling proof that replication studies are an invaluable addition to the scientific record. Indeed, how else would you verify experimental outcomes of a paper? A replication may consume so much time, effort and money that it is not reasonable to expect peer reviewers to do it.
“I also strongly disagree with your attitude that it is “not preferable” for a study to be “merely confirmatory without any novelty.” By this logic, nobody is incentivized to do replication studies. If anything, the current state of certain disciplines and even the mere existence of Retraction Watch itself is compelling proof that replication studies are an invaluable addition to the scientific record. ”
– Preccisely.
@Jonathan Peck thanks for your explanation. Yes I agree I missed that point, because the title and the opening few paragraphs were misleading. That important point (that the authors have not presented any evidence to back up their claim of nonexistence) came too late. It should have been mentioned perhaps much earlier, even in the title of the essay. For example the title should have been this:
“What a database of more than a thousand UNSUBSTANTIATED dismissive literature reviews can tell us”
Indeed any unsubstantiated claim or worse, any dishonest and false claim is malpractice. But dismissive lit review is good and necessary, if done correctly.
—————————————
About your second point, here, you are missing the point. I said “not preferable” and said twice that it is OK. The keyword “not preferable” and especially the keyword “OK but not preferable” does NOT lead to this deduction: “By this logic, NOBODY is incentivized to do replication studies.”
“OK but not preferable” simply means “More workforce and resources should go to new discoveries, while less (less, NOT zero) should be spent on replication studies.”
Also a second point missed by you is that I clearly said when the literature is controversial, confirmatory studies become more important.
After a certain point in this post, the project began to sound like a well-meaning collection of similar articles and more like one academic collecting articles by colleagues whose popularity outsizes their own, possibly out of jealousy.
Academics be academics.
Dismissive sentences are mere peasantries like “how do you do” in English, no? You’re not supposed to take it seriously, and it’s more of just the done thing/etiquette to signal that you speak fluent Academese. It’s more of a byproduct of journal’s addiction with novelty as opposed to relevance rather than the author’s.
“How many of the most ambitious scholars conduct honest research literature reviews”. How many are able to do a literature search competently? As a librarian in a past life I was often shocked at the total inability of many researchers to use a database properly and thoroughly.
When I first started researching the literature for Sales & Operations Planning ( S & OP), I really couldn’t find sufficient literature. Was I being dismissive? No, actually! To prove the lack of S & OP literature, I searched 10 major journals in last 5 years for the keyword ” S & OP”. Out of 9733 articles, I only found 13 articles with the S & OP wordin them. I published the article ‘Systematic Journal Review on S& OP Publications and Avenues for Future Research to Support Smart Industries’. The reason for lack of S&OP literature was because of confidentiality issues associated with order, inventory and capacity data.
Perhaps authors should be required to give the steps used to look for
literature on a given topic, as is done in systematic reviews, when they
claim there is nothing on the topic. Also, when I work with a staff
member who intends to publish, I advise them to say, “to the best of our knowledge, there is not literature…'” if they could not find any
relevant articles.
I really do not like this “to the best of our knowledge” phrasing, which I see used quite often in cases where it seems the author has done a less-than-thorough literature search, and is using it as a cop-out. “The best of your knowledge” should be the state of knowledge of someone who has properly reviewed, and is properly reporting, the literature, in which case using this phrase as a qualification is not necessary. Of course it is always possible to have missed something, but that’s where peer review (or post-publication review) can help (and should be graciously acknowledged). Overall, though, I agree that it could be helpful to ask for ‘dismissive’ reviews to be backed up (e.g. in supplemental data) with a short description of how the literature search was conducted. This might have the bonus effect of improving the standard of literature search by making the methods more salient.
I was likewise confused and did not grasp exactly what the constructed database is trying to accomplish, given that it is largely based on the semantics of generic/filler sentences found in most papers. I do think all these authors could be better writers (myself included, probably). But I can’t find any need or use for a database of papers haphazardly collected around some semantic issues in their introductions.
Of all the diseases ailing academic writing and publishing (plagiarism and LLM-generated papers, fake reviews and reviewers, data manipulation, and various fraud), I would judge “dismissive reviews” at the bottom of the list.
Of course, the “everybody does it” and “it’s just semantics” excuses. Except that not everybody, and probably not most scholars, behave this way. Perhaps I should have listed names of the many scholars I have read who never play these games. But, I didn’t feel it proper to implicate in this mess those who play by the rules, tell the truth, and probably impede their careers in so doing.