A JAMA journal has retracted and replaced a widely circulated 2021 paper which purported to find that pot use among adolescents drops when states make the drug legal.
The article, “Association of Marijuana Legalization With Marijuana Use Among US High School Students, 1993-2019,” appeared in JAMA Network Open and received a bale of attention in mainstream and social media. As GreenEntrepreneur reported:
A September 2021 study of high school use between 1993 and 2019 used the Youth Risk Behavior Survey (YRBS) to determine that adult-use laws did not increase teen use. After two years, states with adult-use laws saw decreases in usage.
But as readers soon pointed out, the findings were schwag. According to the notice:
On behalf of our coauthors, we write to respond to concerns raised about the methods and analyses for the Research Letter, “Association of Marijuana Legalization With Marijuana Use Among US High School Students, 1993-2019,” published in JAMA Network Open on September 7, 2021.1 These concerns were brought to our attention by readers and subsequently published as Comments with the article.2,3 In this study, we used data from the Youth Risk Behavior Survey (YRBS) for the period 1993 to 2019 to provide updated estimates of the association of legalization of recreational marijuana and legalization of medical marijuana with adolescent marijuana use.
The statement then goes into detail about the nature of the error, which, once corrected, led the researchers to conclude:
After reconducting the analyses with weighted and unpooled data, RML adoption and MML adoption were no longer statistically significantly associated with adolescent marijuana use. Moreover, in the corrected analysis, we found that 4 or more years after MML adoption, marijuana use among adolescents declined (OR, 0.83; 95% CI, 0.68-0.99; P = .049), as noted in the Figure in the replacement article.
We apologize to the readers and editors of JAMA Network Open for any confusion we caused by conducting our prior analysis based on unweighted and pooled data when the YRBS recommends that the correct approach is to use weighted and unpooled data.
In an email, which we’ve made available in full here, co-authors Joseph Sabia and D. Mark Anderson said economists and clinicians have different views of how to present weighted and unweighted data:
While applied microeconomists are very nuanced and careful in our academic discussions surrounding weighting, our experience is that physicians are more interested in only showing weighted policy estimates, which we were happy to provide in the revised article.
But the bottom line, they said, was that the conclusions of the paper didn’t change after the weighting:
The above quoted conclusion from our original paper reflects the conclusion of our revised paper. We find no evidence from the YRBS data — whether the estimates are weighted or unweighted, are derived from “pooled or separate” national or state YRBS data, or following economists’ conventional empirical practices or physicians’ preferred approaches — that the legalization of marijuana has (as yet) encouraged youth marijuana use.
We learned a great deal from going through this exercise with the editors, in particular how important it is for researchers across disciplines to be able to talk to one another about technical, nuanced applied econometric issues.
Like Retraction Watch? You can make a one-time tax-deductible contribution by PayPal or by Square, or a monthly tax-deductible donation by Paypal to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].
It seems unfair and, I must say, a little insulting to me to write that “the findings were schwag” when, as the authors point out, this appears to have been a matter of different disciplinary norms around how to analyse and present certain kinds of data.
I agree. I see the attempt to be entertaining by working pot-related references into the article, but I feel it doesn’t fit insofar as the actual issue with the data is concerned.
What are the underlying assumptions, potential pitfalls and also advantages of analyzing these data with ‘weighted and unpooled’ vs ‘unweighted and pooled’ approaches? It would be nice to know this to try to reconcile the differences between the fields of economics & medicine.
Yes. This.
It seems silly to me that a difference between methodology in two fields would make or break a paper so drastically.
In fact, if I were involved, I wouldn’t be looking for a ‘retract and replace’, but rather an update (not sure I would even call it a correction), with the authors adding another section showing both methods of analysis and allowing readers to come to their own conclusions.
I would leave the original paper untouched and just publish the criticism in a letter to the editor and then the authors’ response where they use the other analysis approach and come to the same conclusion.
The first rule of statistics is you’re always doing it wrong in someone’s opinion. But if the effect is real and robust it will show up no matter how you analyse it.
Their findings are also completely believable. When I go out teenagers are not smoking – they are vaping. They can even get away with doing it indoors because it’s low odour. It’s the senior citizens who stink of weed.