Journals typically shy away from publishing data and text readers have seen before — but amidst the newly established norms of open science and data sharing, what counts as a prior publication?
We’re asking ourselves that question after learning that JAMA Pediatrics has rejected a letter rebutting a recent study in the journal about sexual assault on college campuses after deciding that posting the letter on PubPeer is a prior publication.
The submitted letter (which you can read here) was co-authored by independent consultant, therapist and researcher Jim Hopper, who is also a Teaching Associate in Psychology at Harvard Medical School. It concerned a 2015 paper published in JAMA Pediatrics, which suggested that the long-held belief that most rapists on college campuses are repeat offenders may be false. The findings can have major implications for university efforts to stop assaults, as institutions weigh whether to divert resources towards punishment (if serial offenders are largely responsible) or prevention (if most men only commit assaults once).
Hopper has questioned the validity of the authors’ findings, arguing that a problematic methodology clouds the true picture; the authors told us they stand by the paper’s data and conclusions.
Frederick Rivara from the University of Washington in Seattle, editor-in-chief of JAMA Pediatrics, told us:
By disseminating his letter on PubPeer post, this qualified as publication and we don’t publish things already published
That logic didn’t ring true for Hopper, who told us he was annoyed that the journal asked him to squeeze his vast criticisms into a 400-word document, only to receive a rejection in the end.
The phenomenon of rejecting papers that have already been published elsewhere — part of what’s known as the “Ingelfinger Rule” — was originally intended to discourage scientists from publicizing content that was not yet peer reviewed.
The study has received some news coverage, and has so far been cited four times, according to Thomson Reuters Web of Science.
After receiving some of the paper’s initial data from Kevin Swartout from Georgia State University (GSU) in Atlanta, the first author of the paper, Hopper hired an independent consultant, Allison Tracy, to test the paper’s findings.
When Tracy and Hopper discovered flaws in the study, and received what they considered to be insufficient clarifications from the journal and the study’s authors, Hopper informed the research integrity staff at GSU, and was told that they were going to carry out an investigation into the case. A GSU spokesperson declined to comment.
THE BACKSTORY
In recent years, a lot of attention from policymakers has been dedicated to issues around sexual assault on college campuses in the United States — inspired, in part, by the release of the film “The Hunting Ground.”
Many of the policies about preventing sexual assaults on campus have been influenced by the work of David Lisak — formerly Hopper’s graduate school mentor. One 2002 study has been particularly influential, Hopper noted — it showed that out of the 1882 men assessed for acts of sexual violence at the University of Massachusetts in Boston, 120 admitted to carrying out such acts. Out of the group who admitted to committing rape, around two thirds were repeat offenders, who, on average, confessed to committing 5.8 rapes. In total, over 90% of the rapes committed by the 120 rapists were perpetrated by the two thirds who were repeat offenders. The 2002 paper has been cited 92 times.
For years, said Hopper, federal regulators and politicians have relied on Lisak’s work when creating policies around sexual assault on college campuses. He added:
These points have huge implications for prevention and response because if someone comes to you and says they were raped, the odds are that the person who raped them has done it before and will do it again.
But some researchers (particularly those who work in prevention) are worried that the focus on Lisak’s two-thirds and 90% statistics have led policymakers and higher education administrators to conclude that resources should be devoted almost exclusively to identifying and stopping repeat offenders — that is, to law enforcement rather than prevention, said Hopper. He explained:
An argument some have made is that you can have the best prevention programs and roll them out on every campus in the country, but if a man is a repeat rapist, he’s not going to care about those prevention messages and only catching and expelling him or locking him up will solve the problem.
A “SERIAL” PROBLEM?
In the JAMA Pediatrics paper, Swartout and colleagues suggested that those concerns might be founded, since they suggest that most men only commit assault over one time period. The researchers assessed two samples with a total of 1642 participants, and found 172 men responsible for committing at least one rape from the age of 14 to the end of college. They report in their paper:
Most men (67 [72.8%]) who committed college rape only perpetrated rape during 1 academic year.
Hopper, however, criticized the JAMA Pediatrics paper for how its authors defined “serial rapist.” For example, they only counted completed rapes and excluded attempted rape, Hopper said. Another caveat, said Hopper: To avoid double-counting rapes reported on separate survey items — for example an item on rape of an intoxicated person and another item on rape involving anal penetration — the authors reduced their data down to binary rape variables for each assessment period. For each year, they gave each man a “1” for committing at least one rape or “0” for not. But this system can’t account for rapists committing multiple rapes within the same year of college, Hopper added — the only way for a rapist to be labelled as a “serial” offender in the Swartout et al study would be if they received a “1” in different years of college.
Based on his analysis, Hopper said the JAMA Pediatrics data show the opposite trend:
As I’ve shown on PubPeer for the publicly available dataset of theirs, in each year of college, the majority of guys who admitted to committing rape or attempted rape, admitted to doing it more than once — even within every individual year of college the majority were serial offenders who accounted for over 85% of all rapes and attempted rapes.
Tracy, who is based at Wellesley Centers for Women in Massachusetts, released a 75-page technical analysis of the paper’s methodology, and also published a seven-page executive summary of her findings. She told us:
I went into great detail trying to replicate and validate the findings that they report but the more I looked, the worse shape it appeared to be in. In a nutshell, I found technical problems with the data and the estimation of the model as well as inconsistencies in the reported results.
She added that
…even though both the code and the data were provided by Swartout himself, the programming code for creating the analysis dataset did not replicate the analysis dataset.
THE AUTHORS RESPOND
On behalf of all the paper’s authors, co-author Martie Thompson from Clemson University in South Carolina, sent us a statement:
Hopper, Lisak, and Tracy have raised concerns about the Swartout et al. (2015) article that challenges the serial rape perspective on college men who perpetrate persistently across their enrollment in higher education. The paper has been re-reviewed by JAMA statistical consultants and editors. At their direction, we have thoroughly reviewed all of the data analyses, as well as conducted an additional analysis that was requested. The outcome was an editorial decision that the paper is sound, requires no changes, reaches conclusions supported by the data, and that the implications stand as published. We look forward to continuing our studies, future work by other scientists, and collaborating to apply the findings to policy and practice.
That’s not good enough for Hopper, who has asked that the paper be retracted.
But a retraction was never on the table, Rivara said:
We never intended to issue a retraction; [Hopper] thought we should but I never did.
The paper has, however, received a correction — in December 2015, the journal added a note to “Trajectory Analysis of the Campus Serial Rapist Assumption.” It reads:
In the Original Investigation titled “Trajectory Analysis of the Campus Serial Rapist Assumption,” published online July 13, 2015, in JAMA Pediatrics, there were inconsistencies in missing data between the data used for the published analyses and the publicly available derivation data, which affected 2 cases. After correcting for these errors, most of the frequencies and statistics reported in the Results section differ slightly. All interpretations and conclusions remain the same after correcting for these errors. A corrected article with corrections to the Abstract, text, Tables, and Figure has been published online. In addition, this article was previously corrected on August 13, 2015, to fix a column heading in Table 3.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our new daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.
For those interested in my findings on repeat rape perpetration in each year of college, as found in Swartout et al.’s own publicly available data, see http://jimhopper.com/swartout/Hopper_Rape_Frequencies_in_Swartout_Data.pdf
Thanks for doing this analysis. There are a few things that jump at me:
-In slides 12-13 the top and bottom rows are extremely similar, a few pie charts identical. Are those correct?
-I believe slides 19 and 21 have incorrect Y-axis labels, calling things percent where they should be number and vice versa
On the subject at hand….
My reading of the Lisak 2002 data was always different than the authors. They classified a “repeat” or “serial” rapist as anybody committing 2 or more rapes. While that follows standard English language conventions I don’t think that classification makes sense here. Looking at Figure 1, 44 individuals admit to committing rape once, 34 individuals admit to committing rape twice, and 7 admit to committing rape three times (with about 5-10 individuals at each # of rapes for other gradations tested).
The steep dropoff at three suggests (to me) that three or more rapes is a more appropriate grouping for “serial” rapists. The group who committed rape three or more times was of 42/120 = 35%, and was responsible for 371/483 = 77% of all rapes. I would guess that the other violent crimes they measured were much more common in the three-or-more group than the two-or-fewer group.
One thing to keep in mind is that “most rapists are serial rapists” is a very very different statement from “most rapes are perpetrated by serial rapists”. Because of a skewed distribution, where a majority of the crime is committed by a minority of perpetrators, it could be the case that most rapists are not serial rapists, but that most rapes are committed by serial rapists. Most violent crime follows this skewed distribution, and the Lisak data supports this idea. The White 2004 dataset (http://cjb.sagepub.com/content/31/2/182.short) must contain # of acts per person and be able to speak to this question, but it looks like they didn’t report it and apparently the survey data isn’t publicly available.
Thanks for your close reading and questions.
Yes the pie charts on slides 12 and 13 are correct. For some years, assuming complete vs. no overlap of rapes individuals reported made no difference because most or all of the participants reporting rape reported committing more than one rape ONE A SINGLE ITEM. Thus it was not necessary to even consider the issue of overlap to know that those men were repeat rapists.
The White data ARE in the public domain, at least for researchers. And the SPSS code that I used to generate the pie charts on slides 12 and 13, and all of the other analyses presented in my PPT slides, are available via the PubPeer post. In Comment #1 of the PubPeer post, see numbers 5 and 6 under “We have prepared comprehensive documentation to support and flesh out the claims above:”. Those have (#5) instructions for accessing the White (“derivation”) dataset (and identifying the correct variables, as those are mislabeled in the codebook), and (#6) a link to my SPSS code for conducting the analyses used for the pie and bar charts in my presentation.
Also, the White data do include data that can be used to ESTIMATE number of acts per person. As I’ve explained in my presentation, we can only estimate based on assumptions about potentially overlapping rape acts (e.g., a reported rape of an intoxicated woman on one item could be the SAME rape as that reported on another item about oral penetration). That’s why I show everything assuming no overlap and complete overlap (and one could try to estimate numbers in between those two assumptions).
Thanks for catching those y-axis labeling errors (on slides 18 and 21, not 19 and 21). They are now fixed in the online versions of the slides.
On the Lisak and Miller 2002 study, if you dig into their survey methodology, you can see that the number of rapes reported by each man are clearly underestimates for some participants. For example, the Lisak and Miller surveys did not ask the optimal questions to determine how many times the participant committed particular rape acts (e.g., oral intercourse) with how many other adults (i.e., total number of victims times number of times with each victim). I know this because I helped create those surveys and have complete copies of them, and am not saying this to criticize the methodology, as there are always trade-offs and imperfections associated with having to keep surveys from becoming too long and intricate and thus too burdening of subjects. Also, retrospective recall problems alone make such comprehensive estimates problematic.
Similar underestimation issues apply to the White data and even more so for the Thompson data (“derivation” and “validation” datasets, respectively, for JAMA Pediatrics paper). That is, for the White data the response options for each item were: never, once, twice, 3-5 and more than 5. As I explain in my PPT slides, for my calculations I conservatively estimated just 3 for every 3-5 response and just 6 for every “more than 5” response. Clearly the actual numbers are higher, on average.
With respect to the Thompson data, the (2006) version of the Sexual Experiences Survey used doesn’t even allow participants to report frequencies greater than “3+” for any particular item. Unfortunately, Thompson has not published those data, and has refused to make her data available, even though, as demonstrated by the public accessibility of White’s dataset, such data can be shared in ways the honor participant confidentiality and the strictest of research ethics.
Finally, on what constitutes “serial” or “repeat,” that’s a good point. (Keep in mind, however, that men are likely under-reporting such acts of violence on these surveys, for various reasons including retrospective recall bias and reluctance to admit, even anonymously and even to themselves, how many rapes they have actually committed.) Anyone can determine those percentages of how many reported committing one, two, three or more, or more than 5 from the publicly available White data. I’ve run those analyses but not shared them publicly because what I’ve already shared is too much information for most people. Again, anyone could run the frequency analyses using the public data, but here’s what I found using your 3+ rapes = “repeat” rapist definition: For completed and attempted rapes, conservatively assuming 100% Overlap: Pre-college, 46%; Freshman year, 48%; Sophomore year, 57%; Junior year, 69%.
Again, keep in mind these are likely underestimates due to underestimates of numbers of rapes committed. And these are just estimates within SINGLE survey periods / years of college. This raises another key issue that undermines Swartout et al.’s analyses and claims: Is it reasonable to assume, as they have, that men who report committing multiple rapes before getting to college are going to HONESTLY report whether and how many rapes they commit in each subsequent wave of the survey? There is no empirical research supporting this assumption, and common sense and clinical experience suggest otherwise. Yet this is an edifice on which their entire analysis rests (along with the many other problems I’ve documented).
I am curious as to whether the JAMA editors had been initially made aware that some/all contents of the letter had first appeared in PubPeer. I also wonder whether any of the corrections to the Swartout et al., paper were made as a result of the criticisms made by Hopper et al.
Yes, Dr. Rivara was informed in the cover letter accompanying our original letter to the editor (to be considered for publication) that we had posted it on PubPeer, and we provided a direct link to PubPeer post in the cover letter.
Indeed, anticipating this very issue, we explicitly wrote, in that cover letter, “Given the publicity and promotion of the paper, on such a high-profile issue, we have also posted our concerns, including a copy of our letter, on PubPeer.com,” and, “We have not submitted our letter or any other materials for consideration for publication to any other journal.”
In short, we went out of our way to make it clear that the letter was on PubPeer and that we were aware they might consider this “prior publication.” This is all documented in the PubPeer post (see its first and second comments).
Dr. Rivara claims that he “inadvertently missed that on submission,” but I find that hard to believe, given those two paragraphs in our original cover letter. It is also hard to believe that, as editor of the journal, Dr. Rivara would have failed to look closely at my PubPeer post and see the original letter posted there, within an entire paragraph that began, in boldface font, “Letter to the editor of JAMA Pediatrics”.
Finally, it was the executive editor of the entire JAMA Network, Dr. Howard Bauchner, who finally rejected our letter, allegedly on those grounds, after it had been accepted by Dr. Rivara and scheduled for publication. Indeed, Dr. Rivara’s final rejection email to us appears to be copied and pasted from one he got from Dr. Bauchner (complete with mispelling of my name; see 3rd post on PubPeer).
Miguel Roig – Thank you for the good questions.
Jim Hopper – Thank you for answering. You’ve made a very strong case.
The corrections are very small and do not address the full extent of problems with the data that we brought to the authors’ attention and they acknowledged via email — let alone the massive problems with the analyses uncovered by Dr. Tracy. Again, see PubPeer post for details.
I’ve just gone over your posts in PubPeer and while I lack the statistical and subject matter expertise to properly evaluate your concerns, I can only imagine your frustration over how your criticisms have been handled thus far. PubPeer and Retraction Watch (RW) offer plenty of visibility in these cases, but perhaps they only reach a fraction of the relevant research community. Notwithstanding the RW advise you site in PubPeer about contacting ‘the relevant people at the authors’ institution’, and in the absence of an expression of concern over the original paper, I wonder whether you should consider targeting a key journal and repackage and publish your concerns there so that the relevant research community is appraised of the situation ASAP.
I definitely plan to publish some of the findings in a peer-reviewed journal. There are various practical hurdles, including issues with the integrity of the White (“derivation”) dataset and sorting that out, which need to be sorted out first.
I find this all very interesting. As a public employee, I am not allowed to comment on other researchers’ publications, but it seems to me I could comment anonymously via PubPeer without showing my name and affiliation. I also wonder if a LTE to a journal that has a length limitation for letters could include a link to PuBPeer where there is a fuller discussion that allowed by the journal.
During the period our revised letter had been accepted for publication we requested just such a link to PubPeer but the editor rejected that proposal. Also, see the COPE guidelines for how journal editors should respond to critical PubPeer posts (http://publicationethics.org/files/RespondingToWhistleblowers_ConcernsRaisedViaSocialMedia.pdf), none of which were followed by JAMA Pediatrics.
Some clarification on the GSU investigation: Based on information I provided in late December, including in the PubPeer post, Georgia State University conducted a 60-day “Inquiry,” after which (in February) they decided to undertake an “Investigation,” which is ongoing. (I say these things based on explicit, written communication from GSU.)
Also, I only contacted GSU after the chief editor of the entire JAMA Network of journals decided, after our letter had been accepted and scheduled for publication by the JAMA Pediatrics editor, that it would not be published after all.
Bottom lines: I wanted to give the authors, and then JAMA, ample opportunity to conduct appropriate reviews of the evidence and take appropriate action, but when both proved unable or unwilling to do so, I finally turned to GSU as the last hope for doing the right thing.
Very small correction: It was in early March, not February, that GSU completed their 60-day “Inquiry” and decided to launch a full “Investigation” into whether scientific misconduct has been committed by Kevin Swartout.
See my latest comment on PubPeer about how the outcome of GSU’s misconduct investigation is simply not credible, including because the investigation committee’s own statistical modeling experts AGREED with the key findings contained in the report of an independent expert I retained to review the paper’s analyses, using the actual dataset and modeling code provided to me by the paper’s lead author. https://pubpeer.com/publications/26168230
The findings of the independent expert are available in an Executive Summary and 75-page technical report, both linked to from the PubPeer post. And here’s a direct link to her Executive Summary, which concludes, “No reasonable and scientifically grounded debate over the ‘serial (campus) rapist assumption’ can depend on this study.” http://www.jimhopper.com/swartout/Swartout_Critique_Executive_Summary.pdf
The entire thing is nothing short of staggering, and simply bold-faced stonewalling by JAMA.
Moreover, if Pubpeer comments now count as publications, my CV looks fantastic!
Huh – all the astronomy journals allow preprints to appear on the arXiv before/during submission. I don’t believe this was always the case, but today any journal not allowing it would see their submissions drop to zero because the community is willing to act collectively on it.
But the arXiv is of critical importance in astronomy. Papers that don’t appear on the arXiv don’t get read. This gives motivation to engage in collective action. I suppose Pubpeer is not as important?
PubPeer is different. It’s post-publication, and it’s peer review commentary.
I posted my letter there not for pre-publication purposes — getting a letter published was never the point — but rather (1) to contribute to post-publication peer review of the high-profile JAMA Pediatrics paper, and (2) to let people know that (a) the the paper has major problems and (b) JAMA Pediatrics had been apprised of those problems in comprehensive detail and MIGHT conduct an appropriate investigation and have an appropriate response. (There were other reasons too, but those are beyond the scope of your question.)
If publication on PubPeer is going to be used as grounds to reject a paper submitted to a peer-reviewed journal, it follows that publications on PubPeer should be considered citable in the same way as those published in peer-reviewed journals, putting them on a par with such studies.
Lydia, how does one cite an anonymous commentator whose comments are subjected to moderation?
In this case the commentator on PubPeer was not anonymous.
Note, however, that Spectrochim Acta Part A has allowed reference to a PubPeer discussion thread, which includes anonymous commentators https://pubpeer.com/publications/0B89779D4909E2D2C4BCC7D165E2D2
This was in a comment, which can be found here
http://dx.doi.org/10.1016/j.saa.2016.03.013
They don’t have to be anonymous. They can be cited by providing a link. But practical issues aside, I’m just saying that it’s not fair to treat PP posts as publications only for the purposes of rejecting submissions.
I’m also curious about this myself; a post to the topics section of PubPeer could contain original theoretical arguments. Would I then be unable to submit the same to a journal?
I guess if you published anonymously on PubPeer, then submitted the material for publication under your name, you could be suspected of plagiarism, and it would be hard to prove you had a right to the material.