Citation omissions in an economics preprint have set off a wave of recrimination and speculation on a widely read economics discussion board.
Commenters accuse the authors of purposely omitting citations that would have undermined the paper’s claims to novelty and contributions to the field, leveling acrimony and personal attacks. Economists Petra Persson at Stanford and Maya Rossin-Slater at the University of California, Santa Barbara told us they hadn’t been familiar with the omitted papers at the time they first posted their preprint, but their work remains distinct from these previous studies. Nevertheless, the two quickly updated the preprint of their paper – accepted by the top-tier economics journal American Economic Review – to include additional citations. An editor at the journal said it’s not unusual for authors to request such changes before publication, and dismissed the accusations made on the discussion board, calling the site “not a legitimate source of information.”
The study, “Family Ruptures, Stress, and the Mental Health of the Next Generation,” used data from Swedish national databases to compare mental health outcomes of people born to women who lost a relative while pregnant and women who lost a relative in the first year after giving birth. In their manuscript, Persson and Rossin-Slater write that their study is the “first to document a causal link between fetal stress exposure and mental health later in life.” (The economic implications, they note, are that higher rates of stress among poor people may explain why poverty persists across generations.)
The response has been acrimonious, with online commenters accusing the two authors of everything from deliberate omissions to outright plagiarism of study design and datasets. At the Economics Job Market Rumors site, the related discussion thread — which extended well past 1400 comments at the time of this writing — veers from sober analysis toward the offensive. Less constructive comments run the gamut from berating Persson and Rossin-Slater for being successful women in their field to rating them based on their attractiveness. The accusations have extended beyond the two authors to encompass journal co-editor Hilary Hoynes, professor of public policy and economics at UC Berkeley, and four anonymous reviewers whom the authors thank in their most recent draft of the paper.
After the authors initially posted their findings online in December 2015, they have updated the draft with additional citations – including two papers singled out by online commenters, a 2011 paper by Quetzal A. Class and colleagues and a 1978 publication by Matti O. Huttunen and Pekka Niskanen.
In an email, Persson and Rossin-Slater said they became aware of the Class et al paper and others that had studied associations between family rupture and mental health “on or shortly after” May 11, and had uploaded a revised manuscript with additional citations on May 16.
We were not aware of any of these studies when we initiated our research project in 2011, nor when we submitted the finished manuscript to the American Economic Review (AER) in 2014.
In their 2011 paper, Class et al. used an earlier version of the Swedish database that Persson and Rossin-Slater analyzed, and looked at the influence of prenatal stress on infant size and preterm birth risk. Huttunen and Niskanen, in their retrospective epidemiological study, evaluated the effects on schizophrenia risk of losing a father in utero versus losing a father in the first year of life.
In an updated footnote describing Class et al 2011, Persson and Rossin-Slater write:
Related, an epidemiological study by Class et al. (2011) uses Swedish data, and compares the birth outcomes of children whose mothers experienced a close relative death during different months of pregnancy to those of children whose mothers did not. They demonstrate a negative correlation between exposure to death in utero and birth outcomes. However, an important limitation of this study is that it does not fully account for non-random exposure to death, an issue that our identification strategy (and that of Black et al. (2016)) is designed to alleviate. While our primary contribution is to analyze the consequences of antenatal stress exposure on childhood and adult mental health, our results on birth outcomes lend credence to a causal interpretation of these earlier correlational findings.
In another footnote regarding the Huttunen and Niskanen paper, they write:
Huttunen and Niskanen (1978) attempt to filter out the post-natal effects by comparing the mental health of 167 individuals whose fathers died before their birth to that of the 168 individuals whose fathers died in the year after their birth. However, since in utero exposure to the death of a relative affects gestation length (and hence, the date of birth), this approach also suffers from endogeneity concerns.
Persson and Rossin-Slater told us other ways in which their study differs from preceding epidemiological papers:
(i) we focus on deaths of three generations of maternal relatives, (ii) we analyze a larger set of health outcomes, and (iii) we use prescription drug claims data to capture mental health outcomes.
Hoynes confirmed to us that Persson and Rossin-Slater had contacted her to ask if it was acceptable to revise the paper to include the Class et al paper, a request which she granted and described as “not unusual.” Until a manuscript has been published, she wrote, she accepts such changes.
Hoynes also said that she had not heard from any of the paper’s reviewers about the matter, and dismissed Econ Job Rumors as an “unmoderated platform” and “not a legitimate source of information.”
We also received input from Janet Currie, professor of economics and public affairs at Princeton, who was Rossin-Slater’s thesis advisor at Columbia University. Currie echoed Hoynes, saying that Econ Job Rumors is “not a legitimate source of information” and that commenters there:
quite often have the ‘wrong end of the stick’ and unfortunately have threads that harass and malign women in economics…economics is worse than most of the STEM fields in terms of representation of women … perhaps this sort of thing is part of the reason.
Currie confirmed that Persson and Rossin-Slater’s paper uses the same Swedish data as Class et al. did, but that it is data that many authors have used, giving publications by Costas Meghir at Yale and Morten Palme at the University of Stockholm as examples. Many researchers have followed the protocols of the Swedish government for linking to and accessing these data, “and there is nothing unethical or wrong with this,” said Currie. “It just shows how clueless the Econ Job Rumors people can be.”
Currie also said that although Persson and Rossin-Slater initially overlooked some relevant citations, they had already cited several other papers noting a correlation between family ruptures and poor fetal health outcomes:
Economists often take an issue where others have noted a correlation of some sort and try to see whether that correlation can be said to be causal.
We have also reached out to Brian D’Onofrio at Indiana University, the senior author on the Class et al paper at the center of the citation storm. D’Onofrio told us that he is aware of the situation and has emailed the editors of AER, but he did not specify the nature of that email.
D’Onofrio also confirmed he received an anonymous email that has been circulated, which criticized the authors’ decision to update the paper with additional citations. Here is an excerpt from the anonymous email:
It (i) suggests that one can exploit hazy cross-field divisions to plagiarize a research idea without attribution to previous contributions; get an article accepted and (ii) then easily cover it up at printing stage by adding a small footnote when the article is in the galleys and the authors are dealing with typists and admin staff rather than editors.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our new daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.
This is an issue that has been hotly debated in the medical literature since at least the Second Century BC [ME Lewis (2007), Early Empires … p. 165]. How realistic is it to expect a review of the field in a few introductory paragraphs? It’s annoying to hear the umpteenth claim of being the “first to document a causal link between fetal stress exposure and mental health,” but that’s a common kind of scientific puffery which must date back at least as far.
The authors’ defense is weak. First, they claim that they were not aware of the previous work — even though it is the first response in a Google search. In other words, their defense against allegations of plagiarism is they are utterly incompetent at literature review.
Second, their new footnote continues to conceal much of the relevant previous work, and continues to make exaggerated claims of originality.
It is disturbing that Currie tries to muddy the waters by insulting the source of the revelations, instead of addressing the central issue of alleged research malpractice. It is even more disturbing that Hoynes has not sent the paper back out to the referees, and instead expects them to contact her.
Without commenting on the merits or originality of this paper, which is in a field where I have no competence, allow me to point out that these papers are highly ranked by Google search *now* — after they have been at the center of a very controversial matter. How prominently they would have occurred in those results at the time the paper was being prepared is another question entirely, and the answer might well have been very different.
It is fair to ask whether Persson and Rossin-Slater could reasonably have been expected to find these articles before this became a widely discussed issue. I believe the answer is “yes” based on a job market paper from the previous academic hiring cycle:
http://www.sole-jole.org/16208.pdf
In a draft dated October 2015, Tzu-Yin Hazel Tseng showed that she was able to find at least two papers that were mentioned in Econ Job Rumors “on or after” May 11, 2016: Abel et al. (2014) and Class et al. (2014).
It is shameful how Janet Currie tries to discredit the critics by insinuating a sexist motive. The reaction would have been exactly the same with male authors (or editors) involved.
This is the essence of the argument from that message board, as summarize nicely here (note the attribution): http://www.econjobrumors.com/topic/hilary-hoynes-condones-sloppy-literature-reviews-and-cross-field-arbitrage/page/2
“Currie/Hoynes are correct that EJMR is filled with sexists and the AER thread in particular has many, many comments that should have been deleted.
And yet… http://www.econjobrumors.com/topic/new-family-ruptures-aer-nber-is-rip-off-of-obscure-paper/page/5#post-2774524 lists a number of published papers, *still not cited*, which use family death to investigate long term mental health outcomes. Given the Class paper on birth outcomes, as far as I can see, both the “primary and secondary contribution” of the AER paper have already been studied essentially using the same data and same identification strategy (there are minor differences in terms of using deaths post birth as a control, but this strikes me as a minor step which even the authors didn’t claim was their primary contribution).
I should also point out that some of the earlier work more directly considers economic outcomes, and I still have no idea about the basis of the claim in the abstract that “greater stress exposure among the poor may partially explain the intergenerational persistence of poverty,” since this paper and the earlier lit both show *no link* between antenatal stress and any physical health or economic indicators later in life. That is, I still don’t follow what the economics in the AER is supposed to be.”
> There are minor differences in terms of using deaths post birth as a control, but this strikes me as a minor step which even the authors didn’t claim was their primary contribution.
While, I think you have some good points in general, I have to disagree with this claim. The different control group compared to the literature is very important. Eliminating the effect of being raised in a household where a close relative has died is essential to providing an unbiased estimate of the impact of the death being in-vito. The other three of the others lack this control group making this identification method different (better). Also, it appears that the results are different that the other three studies mentioned suggesting that this new control group may of impacted the results.
Assuming that the authors were honestly unaware of the prior them not emphasizing the strength of the control group makes sense since their is not reason to say our study is better than the existing literature. If the authors were aware of the existing work, they would of likely tried a gambit of critiquing the insufficient controls of the existing work.
I tend to assume in these sorts of cases that it is more likely that the authors did only a cursory literature search for non-economics papers and/or handed it over to an RA who screwed up.
There is another paper (1978 publication by Matti O. Huttunen and Pekka Niskanen) that is mentioned here that uses an identically constructed control group as in the AER paper. The identification strategy is almost exactly the same except the AER paper uses expected birth date instead of actual birth date.
It seems that the “not a legitimate source of information” site was the legitimate source of relevant and missing cites.
With all due respect, this is not a case simple “overlooked citation”. The previous studies in question are top results in Google if you use the most basic related keywords. It is either a case of deliberate misrepresentation or gross negligence (i.e. not even doing a google search).
I also find Janet Currie’s “threads that harass and malign women in economics” comment very insulting. It’s absurd to play the gender card when the author of the missed study, Quetzal Class (another woman), has been praised widely for her professional achievements in that same forum thread. Comments like this are insulting to other women like me in the profession.
I am not very impressed by the attempts at proof-by-Google here and above. It is important to know that there are no such things as “top results in Google” or “a Google search”, there are only YOUR top results in Google and YOUR google search. I am largely clueless in both economics and medicine, so the only way I have to know what relevant keywords for, say, the Class et al. paper is to stick in those listed in WoS (stress; pregnancy; timing; preterm birth; low birth weight; small for gestational age). These do in fact bring the paper up fairly highly in various Google searches that I can do (top 5), but where it ends up depends on whether I run a private Firefox session (=tries to get rid of my Google search history), the system I run on (Windows, Mac) and even the web browser (IE and Firefox).
But it is quickly gone from the first two search pages if I remove the words “small for gestational age” (for the reason that this sequence of words is the one that I felt least likely to randomly write when looking for something along these lines, but what do I know), the paper is immediately gone from the first two pages of Google results I get (i.e. around place 25, easily missed). If I instead run them in Google Scholar (which one might naively think should do the trick), I get to do a lot of scrolling ahead, because the paper ends up on about 95’th place, displaced by a lot of much more highly cited papers.
My point here isn’t proving anything about the specific case (which of course I don’t), but to say that Google results are highly volatile. And also to point out that anyone with a potential for of a strong opinion in this case – or any similar case – is likely to also be in posession of the precise keywords and search history to bring up the correct missing citation atop a Google search. It only proves that the reference is central to you, really.
The 1978 paper compared the effects of relative death in utero to death after birth and find a significant difference in mental health outcomes later in life. PP and MRS claim that date of birth is endogenous, so that prior result is not causal. They instead compare outcomes based on the expected delivery date, which according to them will give causal estimates.
Now tucked away in the appendix, they run 2sls where they instrument date of birth with the expected delivery date. They get a first stage R squared of .97, and they mention that “the instrument (relative death before expected birth date) is different from the actual exposure variable (relative death before actual birth date) for only about 1 percent of the individuals in our data.”
What this means is that the endogeneity they are supposedly correcting for may not be that big of an issue. They would get basically the same result just doing the naïve comparison using actual birth date like the researchers did back in the 70s. Interestingly, they do not present the results for this naïve comparison. For all of the fuss that they make about how all the prior literature just gives correlations, they should at least show that their results are different from those correlations.
The main point is that the original paper made a claim of novelty that was simply false (i.e. there is no evidence for humans). It’s not a little false, there are actually dozens of papers that look into this. The authors did not find them and this can happen. We all make mistakes. However, clearly now that we know much more about the paper’s place in the literature, it needs to be subject to another round of refereeing, such that referees can accurately assess the contribution that the paper makes. It’s not difficult to understand this, and it baffles me that the 4 economists in the article above do not get it.
This case and the response of the AER editors (to allow additions after the reviewing stage) creates a very bad incentive for authors to purposefully start hiding related studies at the reviewing stage in order to support their claim of originality only to add them later, after the paper is accepted.
I am not sure if this is common practice in Economics, but in other fields any modifications after the review stage automatically results in another round of reviewing. I think retractionwatch should ask for opinions on the matter from people other than the editor and the authors’ advisor.
There were actually three ways in which they misrepresented their contribution:
1) Pretend that the paper is about economics when actually they are arguing for a medical phenomenon. The connection to intergenerational persistence of poverty is pure speculation, and not even new speculation.
2) Unfairly downplay the existing medical literature showing effects of maternal stress during pregnancy on many aspects of offspring health. These findings already make the point required for the speculation about economics.
3) In particular, fail to cite any of the several papers directly asking about the effects of maternal stress during pregnancy on the kinds of endpoints they consider (birth outcomes and mental illness) using very similar research designs and data.
If they had done the third without the first two, it would look much better for them. Oversights happen. Taken together, the effect is that the way their work is described has almost no relation to its actual contribution. Their updates are inadequate even to address the third misrepresentation.
Even more disturbing is that the reaction to this by third parties like Hoynes and Currie is to shoot the messenger. Researchers of integrity do not normalize misrepresentation behind claims of sexism.
As the Mr. Econ Guy points out, the paper has bigger problems than missing the previous literature. As results from the first stage of the 2SLS regression show, there is virtually no difference between the paper’s finding of a “causal” relationship and the “correlational” relationship in the previous study which the authors severely downplay.
This paper should have been published as a replication exercise in a much lower-ranked journal, after deflating the hugely inflated “contribution to knowledge” which the authors claim. There is no doubt that the authors put in a lot of work and I am inclined to believe that they were careless (and not deliberate) in missing the previous study. But unfortunately, people will rightly point fingers at AER for failing to do a proper job (of course, we have to just ignore any misogynistic trolls) otherwise.
Why do the authors not discuss the Huttunen and Niskanen (1978) in the main text (instead of in a footnote)? The paper looks at the same question with almost exactly the same identification strategy.
Again, people (maybe Petra or Maya or their friends keep making the point) play the moral card and say this is a witch hunt, or sexist, ie that this is personal.
This is not personal. The entire publication system runs on an ethics code of acknowledging prior contributions. This is why we punish plagiarism, fabrication of data or results. Yes there is a grey area in all of these but as shown beyond doubt, we are way past the grey area. Whether it was intentional or not does not matter at this stage; rather what is at stake now is the journal and the editing and refereeing process.
Because what a no punishment outcome validates is a strategy where people just start sourcing ideas, data and regression designs from loosely related fields (demography, health, psychology), and selling it as the ‘first contribution ever’. If it is known that the worst that can happen to me is to have to insert a footnote at some point, then crime starts paying.
What were previously off equilibrium strategies (that would get punished if they got played) now become equilibrium. By backward induction people will stop paying attention to these important cross field areas in economics as they lose their credibility. Which eventually means that there will be less journal space for researchers in these areas. Which means that there will eventually be less faculty positions for these areas.
In the long run this behaviour of the AER is putting into question the only thing we run on: credibility. And it is doing so particularly for people in these cross over fields like health Econ and the like.
This is what is at stake here, not Petra or Maya.
It is fascinating how no top economists have voiced their concern in the fear of being branded as misogynist. RW should contact the leading economists and ask for their opinion. EJMR may not be credible source of information (though it was credible enough for these authors to update their paper based on the information there, since they cannot magically find the papers that they didn’t manage to find in four plus years), but surely someone like Prof Heckman or Prof Card will be a good source. The cronyism has been going on for too long, and should be stopped if we want to have any semblance of scientific curiosity alive in this discipline.
Having briefly skimmed the thread and some of the pages at random I can’t say that this provides an accurate description of what has been written there.
“The response has been acrimonious, with online commenters accusing the two authors of everything from deliberate omissions to outright plagiarism of study design and datasets. At the Economics Job Market Rumors site, the related discussion thread — which extended well past 1400 comments at the time of this writing — veers from sober analysis toward the offensive. Less constructive comments run the gamut from berating Persson and Rossin-Slater for being successful women in their field to rating them based on their attractiveness. The accusations have extended beyond the two authors to encompass journal co-editor Hilary Hoynes, professor of public policy and economics at UC Berkeley, and four anonymous reviewers whom the authors thank in their most recent draft of the paper.”
The focus in this paragraph is mostly on the sexist comments which undoubtedly makes good copy and fits nicely into a story that has been, rightfully, told many times, but it is hardly representative of what has been written there. The whole section from “Less constructive comments … draft of the paper.” could have been left out.
A proper description would ferret out the claims made in that thread and describe these. In addition, the claims made by the authors and quoted here would be compared to the claims made in said thread. This, however, has not been done. Fortunately, the comments seem to address this somewhat.
Now I can understand not wishing to read through all 1400+ responses to that thread, surely you have better things to do with your time, but to misrepresent the content of the thread and fit it into a nice narrative is doing the readers a disservice. I’d like to have seen more balance here.
To be frank, the AER’s representative Professor Hoynes has not addressed the substantive issue here.
As pointed out by Mr. Econ Guy, the first stage of the 2sls shows that the causal estimates presented by the authors are likely no different from the correlational estimates they claim to be improving upon. A simple Hausman test could resolve this issue, but it does not appear that the authors performed one.
Secondly, the authors propose using the expected date of birth to solve the endogeneity problem of the actual birth date (if there is actually a problem).
Their justification is: “To address these issues, we adjust our treatment variable by defining it relative to the expected date of birth at full term instead of the actual date of birth. More precisely, we define a child’s estimated date of birth as e_b = c + 280, that is, 280 days (40 weeks) after the date of conception, c. Unlike the actual date of birth, this expected date of birth is pre-determined at the relative’s death date.”
Certainly the conception date is pre-determined relative to the death, but how do the authors know the conception date? The conception date is calculated based on the gestational age of the baby at birth (conception date=birth date-gestational age). However, the gestational age is itself an estimate based on the last menstrual cycle or measurements taken from a prenatal ultrasound. The prenatal ultrasound is the preferred method for estimating gestational age and is used if it gives a significantly different answer from the estimate using the last menstrual cycle. Also, and this is the key point, the accuracy of the estimate of gestational age based on ultrasound decreases with actual child age.
So conception is predetermined, no argument there. But is estimated conception date predetermined? If pregnant individuals happen to miss early prenatal appointments because say a close relative dies, then the estimate of gestational age is affected and the estimated date of birth is not exogenous. In the paper, they happen to find that mothers who experience death of a close relative are less likely to have adequate prenatal care (which they claim corresponds to roughly one missed prenatal visit). They argue that the magnitude is too small for lack of prenatal care to explain their outcomes (never mind that the magnitude of most of their estimates are tiny), but it could easily affect the estimate of gestational age.
Some of their results are consistent with they hypothesis that mothers who experience death of a relative while pregnant have poorly measured gestational age. This can be seen in the increased probability of pre-term birth and the increased probability of C-section. Since the C-section right censors outcomes, this can lead to the decrease in average birth weight and size measurements that they estimate. Probability of low and very low birth weight is higher and those thresholds are independent of the gestational age estimate so those effects are probably “clean”. Importantly, their finding of the increase in probability in preterm birth is what leads them to conclude that actual date of birth is endogenous for the purposes of looking at later life outcomes.
In summary, the authors have not established that endogeneity biased prior estimates in the literature (estimates, by the way, whose existence they only reluctantly acknowledged). They have also not shown that their measure is not endogenous. All of their tests for exogeneity rely on the conception date being known. I would like for the editor and authors to respond to my clearly sexist comments in the most dismissive and insulting way possible.
Anyone who thinks that sexism is not a part of this story is, in my opinion, incredibly naive.
It is possible to play the “missed citation” game with almost any paper. Yes, there are varying degrees of oversight, but the sad reality is that many men in the profession have a tendency to discredit research by female economists. (Justin Wolfers has made this point eloquently and repeatedly in his NY Times columns.)
The existence of this thread on Econ Job Market Rumors, and the incredible response to it, is at least in part due to economists’ proclivity to undermine the research of women. This is, I suspect, what Profs. Hoynes and Currie meant by EJMR not being a “legitimate source of information.”
Any anonymous and mostly unmoderated forum will contain unsavory elements such as racism, sexism, xenophobia, etc. Econ Job Rumors is no exception.
However, dismissing ALL posts on Econ Job Rumors because SOME posts are sexist is clearly a case of the ad hominem fallacy. As others have pointed out above, it is ironic that Currie and Hoynes both dismissed Econ Job Rumors as illegitimate when all the missing papers that were added “on or shortly after May 11” were discussed on Econ Job Rumors.
No doubt it is. It is, however, not a big part of the story. I would be surprised if the quantity of sexist and misogynistic posts would exceed the 5-10 % range or motivate more than 5 % of those posting in that thread. Perhaps I’m wrong, but this is something that can quite easily be checked by those making this particular claim and by those who are poisoning the well.
Shouldn’t this be judged on the merits rather than unfairly stereotyping and dismissing the views of individuals because they happen to be economists? It seems rather ironic don’t you think?
I haven’t quantified the EJMR posts, but you are probably right that only a small fraction of them are blatantly sexist. I disagree, however, that only a few are motivated by sexism (nor is this easy to verify). I suspect that a lot of the interest in this story stems from the fact that the authors are young, successful women. It’s difficult for me to otherwise explain how a few missing citations would create such a stir.
You know that isn’t true. It’s not a case of a few missing citations. It’s a case of missing very similiar, almost exact studies, in order to inflate the contribution of the paper.
You write, “I suspect that a lot of the interest in this story stems from the fact that the authors are young, successful women. It’s difficult for me to otherwise explain how a few missing citations would create such a stir.”
I present a counterexample: http://www.econjobrumors.com/topic/lognaics-is-a-scandal-that-everyone-is-simply-ignoring
The individual at the center of that thread is not a young, successful woman. Rather, he is a young man. If you believe a lot of the interest in this story is based on sexism, then please explain why 86248 views, 994 posts, 50 pages were devoted to a discussion about a young man?
How much interest would you have expected there to be in the story if the article had been written by two successful young male economists? That would be the ideal counterfactual, right? If there would be a substantial difference, then we would probably agree that this could be attributed to the gender of the authors.
Sadly, or perhaps fortunately, we don’t have such a counterfactual. However, I find it difficult to imagine that such a hypothetical scenario would attract any less attention. If past scandals in economics and elsewhere are anything to go on, then it seems that a scandal is a scandal and it will attract lot’s of attention because of that as long as it’s visible enough. Now perhaps two young females are more visible than two young males due to the demographics of the profession, but then the cause, I’d say, is rarity and not sexism.
In any case, I think that we both do agree on that the case should be looked on based on the merits and not how much attention it has been receiving. And in terms of merits, the comments made, there and here, seem to indicate that it is about considerably more than a few missing citations. Granted, I’m not an expert, but this is also why I’m so very disappointed by the focus on sexism instead of on comparing the claims made by the authors and the claims made on said messageboard about the article written by the author and the articles written by others. The focus on sexism seems vastly out of proportion compared to the quantity of sexist posts. This, I believe, we agree about.
Based on my experience writing and reading referee reports, it seems far-fetched to think that this paper’s outcome would have been different had the original submission included a couple of additional citations to papers in medical journals–one of which is 30+ years old with a sample size of under 400. This is what I meant by “a few missing citations.”
We can certainly debate whether the system for choosing top publications is appropriate. But if the real issues are elitism and preferential treatment–as others have claimed–I think it is unfair that the authors of this paper will bear the brunt of the professional (not to mention personal) criticism. The fact that these women have become scapegoats is what strikes me as sexist.
“Based on my experience writing and reading referee reports, it seems far-fetched to think that this paper’s outcome would have been different had the original submission included a couple of additional citations to papers in medical journals–one of which is 30+ years old with a sample size of under 400.”
Your posts have a pattern of being heavy on assertion and light on arguments supporting the assertions. It is hardly far-fetched to think that this paper’s outcome would have been different had the original submission included the numerous missed citations to papers in medical journals.
You single out the Huttunen and Niskanen (1978) as a paper whose citation would not have made a difference. Persson and Rossin-Slater differentiate their paper from Huttunen and Niskanen by saying that “since in utero exposure to the death of a relative affects gestation length (and hence the date of birth), this approach also suffers from endogeneity concerns” (footnote 10).
Yet, in footnote 24, they say that “the instrument (relative death before expected birth date) is different from the actual exposure variable (relative death before actual birth date) for only about 1 percent of the individuals in our data.” That is, the “endogeneity concerns” are trivial at best.
This is shown further in table D1 in the appendix of Persson and Rossin-Slater. The first stage coefficient of regressing the “endogenous” relative death before actual birth date on the “exogenous” instrument ranges from 0.971 to 0.973. Thus, Persson and Rossin-Slater’s paper essentially amounts to a minor extension of a paper from 1978.
I find it interesting that both Hoynes and Currie used the same awkward phrase “not a legitimate source of information.”
I am appalled to see the sexism card played in this case because 99% of the comments in the thread are indeed substantive and explain why the paper is a pure replication and present the indignation of the profession regarding how the authors and the editor handled the case when the new studies were revealed. I am a female economist who is working hard to promote females in the profession and I believe that there is definitely plenty of sexism in economics. However, in this case, the issue is about the fact that when you are well connected it seems that there are no repercussions to your actions and this is regarding one of our top two economic journals, which makes and breaks careers. The issue is way bigger and more important than what this two authors did and the editor in charge is equally or even more (as more experienced) to blame as the authors.
The issue is not merely “a few missing citations.” The first issue is that the contribution of the paper is much more modest than that claimed in the version that was peer reviewed. The second, much larger, issue is how the editor responded to the new information that the contribution is much smaller than claimed.
There is no evidence that the serious criticism is motivated by sexism, and if we insist on going down that road, the prima facie case would be that the female editor and the female supervisor of one of the authors are protecting the female authors. But there is also no evidence that that is the case, and my prior is that that’s not credible.
Many more comments have expressed concerns over preferential treatment of well-connected authors at leading economics journals than have expressed anything sexist, and that seems to me to help explain why there is so much interest in this case. I have little doubt that if the authors were merely rank-and-file professors: (1) the paper would probably have been desk-rejected at the AER and correctly sent to a field journal; (2) if the paper had made it to review at the AER, the editorial decision would have correctly been `not enough of a contribution, send it to a field journal’ and (3) if somehow the paper had been reviewed and accepted at the AER and then found to have a much lower contribution than claimed, the editor and other powerful economists would not have circled the wagons.
The claims by defenders of the paper that this is an issue of a few missing citations are inaccurate and unhelpful. If this were a paper making a significant contribution that simply failed to note some of the relevant literature, the furor would have died down days ago when the citations were added.
The issue is that
1. The paper makes no significant contribution to the literature,
2. The papers left uncited were precisely those that addressed the exact same questions, making this paper little more than a replication on a topic that is well-trodden ground in the medical literature, and
3. In light of 1 and 2, the paper clearly doesn’t represent a contribution worthy of AER, but no action is being taken to e.g. assign new referees and reassess its acceptance.
Additionally, while the focus by defenders of the paper and this article has been on a gender-based motive for this controversy, surveying the thread makes it clear that most of the vitriol is directed at elitism (perfectly captured in Currie and Hoynes’ condescending responses), nepotism, and clubbiness in the economics profession. People are angry because this is merely an unusually clear-cut example of well-connected researchers leveraging their professional networks to publish low quality work at the expense of less connected scholars, a longstanding source of frustration in the profession, and this paper is still being pushed forward and aggressively defended even in light of overwhelming evidence that it does not belong in a top journal.
Improved version of real abstract of this paper:
“Recently it has been documented that pre-natal maternal stress is associated with low indices of fetal development (Class et al, 2011). Further, it is also known low indices of fetal development are correlated with later psychological and cognitive problems (see Huttunen and Niskanen, 1978 for an early statement and D’Onofrio, Class et al, 2014 for a recent review), suggesting that pre-natal maternal stress is associated with long-term psychopathologies.
In this paper we assess whether Class et al (2011) findings are robust to a change in regression design. In particular, sourcing from a well known strategy in the pre-natal maternal stress literature (that used, for example, in Huttunen and Niskanen, 1978), we propose that the control group should be constituted by off-spring subject to immediately post-natal maternal stress. We confirm the earlier findings in of Class et al (2011) and suggest these are likely causal. In order to ensure comparability across the two studies, we use the same Swedish population data. As a secondary contribution, we extend the range of outcomes associated with pre-natal maternal stress treatment”.
That is:
1) the idea is not new
2) the application of this idea to this particular data set is not new
3) the particular identification strategy is well known and dates back to at least 1978
Precisely, what you are describing is a replication study and not to mention that using an instrument doesn’t really change the conclusion of the “correlation study” by Class et al (2011). So do you think with this knowledge this paper would have gotten into AER? I highly doubt it! I give the referees the benefit of the doubt that with the right knowledge of the background literature they would have rejected it as it has been done numerous times for other papers for the same exact reasons. Maybe the issue is that health science papers with very LITTLE economics as this one should not be in economics journals as the referees don’t know the medical literature and for that the blame is on the editors. However, to call it missing a few citations is to distract from the big issue. This paper shouldn’t be AER and a good editor should have re-sent the paper to referees after knowledge of the new studies emerged. I believe many people would have been happy with this course of actions including myself. However, instead they chose to play the gender card.
Professors Hoynes and Currie should respond here.
up to now I had mixed feelings about this story. I understood the missed citations, but also that maybe it was too much to accuse the authors of plagiarism (and still think so).
However, I have now checked Hilary Hoynes’ CV here:
https://gspp.berkeley.edu/assets/uploads/faculty/cv/Hoynes_cv.pdf
and found
“The Long Run Effects of Great Society Programs” (with Martha Bailey, Maya Rossin-Slater and Reed Walker)
SO THE EDITOR IN CHARGE OF THIS SUBMISSION IS ALSO A COAUTHOR OF ONE OF THE AUTHORS OF THE RAPTURES’ STUDY (MAYA ROSSIN-SLATER)?
Isn’t this prohibited?!?!
(“There are several rules that affect assignment of manuscripts. Coeditors are generally not assigned manuscripts authored by an individual at his or her institution, by an individual with whom the Coeditor has been a recent coauthor, by an individual who has a close professional or personal relationship with the Coeditor, or by an individual who has served as a graduate student advisor or advisee of the Coeditor. Papers falling into these categories are handled by the Editor or by a different Coeditor with appropriate procedures for confidentiality of refereeing. Papers submitted by a Coeditor are handled by the Editor and papers submitted by the Editor are handled by a Coeditor, again employing appropriate confidentiality procedures.”
https://www.aeaweb.org/journals/aer/about-aer/editorial-policy)
I know that it is difficult to convey this credibly in this context, but rest assure that I got a PhD from an Ivy league university and I am currently a tenure-track faculty member in one with several top publications. So I am not holding “the wrong hand of the stick” as claimed quite arrogantly by Janet Currie.
@RetractionWatch: this needs to be made an issue. There is now direct evidence that a co-editor handled the peer-review process of a paper of a recent co-author, which violated AER policy and is clearly unethical. This needs to be made public so that there will be consequences.
Unbelievable – I am not an economist but I am an academic in another discipline. This sort of thing would never happen in my field and I would hope something is done to fix this – for the sake of taking economics seriously as the science it is
Even more scandalous is that the editor (Hoynes) is co-authoring at least two studies with on one of the authors of this study. How can she credibly defend the paper? This is total corruption.
All of this is conjecture.
This is a valid point. That is why I mentioned a job market paper from the previous academic hiring cycle:
http://www.sole-jole.org/16208.pdf
In a draft dated October 2015, Tzu-Yin Hazel Tseng showed that she was able to find at least two papers that were mentioned in Econ Job Rumors “on or after” May 11, 2016: Abel et al. (2014) and Class et al. (2014). So it was reasonable to have expected Persson and Rossin-Slater to find at least some of the overlooked citations.
The American Economic Association should issue a statement either justifying why an editor was allowed to handle a paper of their collaborator in this case, or acknowledge that this was a deviation of their rules and begin an investigation. If they do neither this case will fester and tarnish the reputation of the flagship journal of the economics profession.
This whole fiasco with Hilary Hoynes and Petra Persson is a great blackeye to the AEA.
Doesn’t the fact that none of the four referees of this paper were aware of the prior literature on the subject (or were suspicious enough that such a literature might exist to google it) draw into question their qualification to referee a paper on this topic?
It seems to me that that is the big question. It seems entirely possible to me that the authors didn’t investigate the literature in other disciplines but if you are going to publish a paper on the effect of X on Y shouldn’t you send it to someone who knows something about X, Y, and their potential relationships? The ever expanding set of things economists think they should study could be not just embarrassing but dangerous.
Howver, I’m not sure that this paper should go back out to referees; I don’t see why Persson and Rosslin-Slater should bare the burden of what should be a re-examination of AER’s publication process on non-economic topics.
Classic case of imperial science, really.
I cannot disagree with the substantive reasons why people object to this article. However, such an approach has been a norm in the profession and it is wrong. The profession’s peer review system requires a grand reform. However, what is puzzling more is — why people keep silent until a minority is replicates the strategy of the established elite and use the case against a minority? It is respectable to call for a system wide reform. It is not respectable to selectively adhere to scientific standard.
Your premise is flawed. Anger has been building in the community for some time, independent of the “minority status” of the offenders involved in the other recent cases. This particular case was simply so egregious that it was the last straw. The minority angle is 99% irrelevant here.
https://m.soundcloud.com/panoply/accountable-care-organizations-more-swedish-administrative-data-and-the-big-city-recovery
Start listening at 40:00
I should have linked directly to Vox
http://www.vox.com/2016/6/3/11852934/the-weeds-economic-recovery-cities
A group of anonymous economists have issued a response.
ABSTRACT
Persson and Rossin-Slater (2016b) claim to provide the first causal estimates of the effects of fetal stress exposure on mental health later in life. They emphasize that their analysis is the first to address non-random exposure to a relative’s death and the endogeneity of gestation length to fetal stress. In light of discoveries regarding prior literature, we find these claims to be exaggerated and misleading.
https://mpra.ub.uni-muenchen.de/71699/
_________________
Why the anonymity? Why a pseudonym (a play on Nicolas Bourbaki)? Frankly, it’s for fear of professional retribution.
“The economists value journal publications highly and have a clear preference among them, with the American Economic Review (AER) the most preferred. Their responses imply they would sacrifice more than half a thumb for an AER publication.”(1)
“Currently about 40% of top-five publications are in The American Economic Review, giving editors of this journal potentially substantial power to shape the future direction of economics as well as the careers of particular economists.”(2)
Citations
(1) Attema, Arthur E., Werner BF Brouwer, and Job Van Exel. “Your right arm for a publication in AER?.” Economic Inquiry 52.1 (2014): 495-502.
(2) Gibson, John. “No Top Fives, No Worries? | VOX, CEPR’s Policy Portal.” No Top Fives, No Worries? | VOX, CEPR’s Policy Portal. 6 June 2014. Web. 04 June 2016.
There have been several questions raised about the ethics of the labor economics editors at the American Economic Review. The other areas of economics don’t seem to be in the spotlight as much.
The most important question is yet: Why is such a study published in an economics journal? But that seems to be a common trend.
The authors have now included a non-refereed “post acceptance” note acknowledging that the actual contribution of the paper is much more limited than initially claimed: http://www.econ.ucsb.edu/~mrossin/Persson_RossinSlater.pdf
According to the manuscript, the coeditor in charge and coauthor of one the the authors has taken the responsibility to accept the new version. The same editor that stated few weeks ago: “Hoynes also said that she had not heard from any of the paper’s reviewers about the matter, and dismissed Econ Job Rumors as an “unmoderated platform” and “not a legitimate source of information.”
Two senior economists have now come out decrying the process and the AEA’s failure to address the issue publicly thus far: https://gborjas.org/2016/06/30/a-rant-on-peer-review/
The key issue for me about this whole episode is how economists shamefully avoid looking for earlier studies in Web of Science, Google or wherever. We see this time and time again. Psychologists can be guilty of this and other disciplines also. But these elite economists have absolutely no excuses for not doing literature searches – building on the shoulders of … etc. This is the FIRST thing we all learn as undergraduates.