The National Football League failed to include data from diagnosed concussions in peer-reviewed studies, making the sport look safer than it is, allege the results of an investigation published yesterday in the New York Times. Now, the paper and the NFL are arguing over whether the studies were supposed to include every instance of head injury.
Early studies on concussion rates published in the journal Neurosurgery left out at least 100 instances of of concussions, the Times reported. The Times and the NFL disagree on the implications of studies based on an incomplete data set: Sources told the Times that it’s bad science, while the NFL explains that the studies were “necessarily preliminary.”
Yesterday afternoon, the sports league published a statement saying that the Times story “is contradicted by clear facts” and “sensationalized.” The statement argued that:
The studies never claimed to be based on every concussion that was reported or that occurred.
In a reply to the NFL statement, the Times says:
The studies and peer-review statements did, in fact, claim that.
In response to the NFL’s statement, the Times also notes:
The N.F.L. statement said, “Moreover, the fact that not all concussions were reported is consistent with the fact that reporting was strongly encouraged by the league but not mandated, as documents provided to The Times showed.” At least one of the papers said it was, in fact, mandated.
The Times piece explains:
For the last 13 years, the N.F.L. has stood by the research, which, the papers stated, was based on a full accounting of all concussions diagnosed by team physicians from 1996 through 2001. But confidential data obtained by The Times shows that more than 100 diagnosed concussions were omitted from the studies — including some severe injuries to stars like quarterbacks Steve Young and Troy Aikman. The committee then calculated the rates of concussions using the incomplete data, making them appear less frequent than they actually were.
After The Times asked the league about the missing diagnosed cases — more than 10 percent of the total — officials acknowledged that “the clubs were not required to submit their data and not every club did.” That should have been made clearer, the league said in a statement, adding that the missing cases were not part of an attempt “to alter or suppress the rate of concussions.”
One member of the concussion committee, Dr. Joseph Waeckerle, said he was unaware of the omissions. But he added: “If somebody made a human error or somebody assumed the data was absolutely correct and didn’t question it, well, we screwed up. If we found it wasn’t accurate and still used it, that’s not a screw-up; that’s a lie.”
These discoveries raise new questions about the validity of the committee’s findings, published in 13 peer-reviewed articles and held up by the league as scientific evidence that brain injuries did not cause long-term harm to its players. It is also unclear why the omissions went unchallenged by league officials, by the epidemiologist whose job it was to ensure accurate data collection and by the editor of the medical journal that published the studies.
We’ve done a preliminary scan of the research, and present below a series of 16 papers from the journal Neurosurgery that match the time-frame and subject matter discussed in the Times piece:
- Concussion in professional football: reconstruction of game impacts and injuries, cited 243 times, according to Thomson Reuters Web of Science
- Concussion in professional football: location and direction of helmet impacts-Part 2 cited 124 times
- Concussion in professional football: epidemiological features of game injuries and review of the literature- Part 3 cited 109 times (referenced in the NFL’s response)
- Concussion in professional football: repeat injuries–part 4 cited 49 times (referenced in the NFL’s response)
- Concussion in professional football: injuries involving 7 or more days out–Part 5 cited 55 times (referenced in the NFL’s response)
- Concussion in professional football: neuropsychological testing–part 6 cited 83 times
- Concussion in professional football: Players returning to the same game – Part 7 cited 48 times
- Concussion in professional football: biomechanics of the striking player–part 8 cited 57 times
- Concussion in professional football: brain responses by finite element analysis: part 9 cited 78 times
- Concussion in professional football: comparison with boxing head impacts–part 10 cited 67 times
- Concussion in professional football: helmet testing to assess impact performance–part 11 cited 45 times
- Concussion in professional football: recovery of NFL and high school athletes assessed by computerized neuropsychological testing–Part 12 cited 97 times
- Concussion in professional football: performance of newer helmets in reconstructed game impacts–Part 13 cited 50 times
- Concussion in professional football: biomechanics of the struck player–part 14 cited 82 times
- Concussion in professional football: animal model of brain injury-part 15 cited 23 times
- Concussion in professional football: morphology of brain injuries in the NFL concussion model-part 16 cited 18 times
A critic argued to the Times that the NFL’s incomplete data set was not good science:
“One of the rules of science is that you need to have impeccable data collection procedures,” said Bill Barr, a neuropsychologist who once worked for the Jets and who has in the past criticized the committee’s work.
By excluding so many concussions, Mr. Barr said, “You’re not doing science here; you are putting forth some idea that you already have.”
Recently, we reported that critics of a 2008 study about the effects of concussions failed to note their ties to the NFL.
The Times notes that the NFL’s conduct resembles that of another industry — Big Tobacco:
Some retired players have likened the N.F.L.’s handling of its health crisis to that of the tobacco industry, which was notorious for using questionable science to play down the dangers of cigarettes.
Concussions can hardly be equated with smoking, which kills 1,300 people a day in the United States, and The Times has found no direct evidence that the league took its strategy from Big Tobacco. But records show a long relationship between two businesses with little in common beyond the health risks associated with their products.
In a letter to The Times, a lawyer for the league said, “The N.F.L. is not the tobacco industry; it had no connection to the tobacco industry,” which he called “perhaps the most odious industry in American history.”
Still, the records show that the two businesses shared lobbyists, lawyers and consultants. Personal correspondence underscored their friendships, including dinner invitations and a request for lobbying advice.
The NFL responded to the alleged connection:
As the Times itself states: “The Times has found no direct evidence that the league took its strategy from Big Tobacco.” Despite that concession, the Times published pages of innuendo and speculation for a headline with no basis in fact.
The NFL statement concludes:
The Times‘ sensationalized story is further refuted by the NFL’s ongoing commitment on the issue of player health and safety — notably, to the support of research, including that of our most vocal critics, on the long-term effects of concussions in all sports, and to change our game in an effort to make the sport of football as safe as it can be. We have committed tens of millions of dollars to fund independent research, made 42 changes to our rulebook since 2002 to make the game safer, and have advanced concussion awareness and safer tackling at all levels of the sport. And we provide a host of benefit programs which, together with the proposed settlement of our players’ concussion litigation, will ensure that our retired players are properly cared for in the future.
Contact sports will never be concussion-free, but we are dedicated to caring for our players, not just throughout long careers but over the course of long lives.
The Times tried to reach Michael L. J. Apuzzo, the editor of Neurosurgery when the papers were published, but he did not respond to interview requests.
We have reached out to the current editor in chief of Neurosurgery, and will update this post with anything else we learn.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our new daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.