Another installment of Ask Retraction Watch:
I reviewed an article for two different journals that presented data from a large non-public data set. A previous publication from the same group had presented findings on the same topic from the dataset, but the new paper didn’t mention these previous analyses. The new paper had more detailed analyses. As a reviewer, both times I said that they really needed to mention that there were previous analyses of the same topic from the same dataset and say what their new analysis was contributing (not much!). Both times they basically refused and it got rejected. Then it got published in another journal (I didn’t review it this time) still without citing the previous analyses from the same data set.
Take our poll, and comment below.
[polldaddy poll=7479318]
This is frustrating but I try to keep in mind that (a) there are so many obscure, n-th- tier journals where you can publish just about anything and (b) even in the reasonable-to-good journals peer review is a somewhat statistical process and if you try enough times (assuming that the paper isn’t completely bad) you may tunnel through an energy barrier that you wouldn’t otherwise cross in a classical mechanical sense.
Who knows how common this phenomenon is. This time the reviewer caught by mere accident!
One fundamental here is the thoroughness of the peer review. Theoretically the review process, by multiple peer reviewers, shoud prioperly vet a manuscript. But, we scientists lead busy lives, and if we are altruistic enough to accept a reviewing task, that task squeezes our “free” time even more. So there is a temptation to do cursory reviews. (I, “masochisticaly,” do not, esp., with papers from certain parts of the world, and from researchers in my field who are unknown to me.) I have seen cursory reviews when I read the ‘other reviews’ or, as a subject editor, compare reviews. Here are the holes through which abuse creeps.
I would surely contact the editor (most likely anonymously), and consider writing a note on this paper on Pubpeer. I think reviewers are supposed to be confidential, and not reviews. This notion of hidden reviews has poisoned the peer review system and now each author always finds some way of getting anything published and not being criticised on this.
I’m not so sure about hidden reviews “poisoning” the peer review system. If that were a problem, then I would say a bigger one is that a single person (the editor) has the final say. In some journals, the reviewers are just giving recommendations to the editor. The editor selects the reviewers and then makes a decision based on the reviews received — that’s a lot of power for one person.
Sometimes, it might be better for reviewers to be selected “at random” from a list of people in the field and then each assign a score from 0 to 100 and then averages it.
Opps! Sorry, didn’t mean to digress and hijack your original comment…
I see your point, and actually I think it makes mine even stronger — it is a lot of power on a single person to decide whether a paper will be published based on hidden arguments. Actually what I find most bizarre is the growing habit of journals of requesting authors to select their own reviewers. This leads to fake reviewers, one`s mama, bang buddy, etc, accepting crappy papers in 10 mins…
I voted for the reviewer to contact the editor of the journal that published the paper. However, I do wonder what the reviewer’s options are should the editor of that journal chooses not to take any action on the matter. In part to avert this scenario, perhaps the reviewer should also cc’s his/her concerns to the editor of the journal that had published the earlier paper.
“I do wonder what the reviewer’s options are should the editor of that journal chooses not to take any action on the matter” — I think the best is to inform readers at Pubpeer, maybe even including a copy of the review sent to authors. I think public disclosure is the best solution for most problems in Science.
Agreed!
I have an ethically related situation right now: I helped a group to get into my field, I was happy to see a manuscript, I tried to correct it – and I then found a fundamental error in the experimental setup. I pointed out how to repeat the experiment without the accident and was sure another author who should get only the final corrections would be somehow informed about my finding of the wrong experiments. Unfortunately, the manuscript got submitted and published (without me), but I have no idea how the reviewers could overlook this and how the other author could be tricked to remain a coauthor. I think I should also inform the Editor of the quality of his reviewers. I think it is OK to mention in the above case the potential wrongdoings of the authors. It is against scientific standards to publish the same data twice; in this case, it appears to be that the authors had a reason to hide the older publications on the same data. I wonder about the journals policies to retract the paper or to add a corrigendum.
I am not entirely clear on the exact issues of your case (in addition to a fundamental error in method and possibly the covert reuse of data, is there a question of authorship as well?), but it seems that you should follow a similar approach as suggested in the case outlined above: Provide a detail description of your concerns to the editor of the journal (both journals if data were covertly reused) and consider doing so in Pubpeer as well.
I’m not sure why the confidentiality of the review process is an issue. As I understand it, the objection to the paper is based on facts that are now entirely public.
Why ? I think only the reviewer knows of the correlation between the data in the specific case. And why should a paper be judged based on confidential opinions?
That wasn’t how I read the description, but perhaps that is what was meant.
This is salami slicing at its worst and one could argue there is a degree of misconduct, since the previous analysis was the authors’ own, so they must have known about it. So there is nothing wrong with re-analysing data, but the previous analysis must be cited and indeed discussed in the light of the new one.
By all means contact the journal, but few editors will take any sort of action.
Pubpeer is the answer. You can either put your review up there and/or comment that the paper fails to cite the previous analysis.
Once the article is published, the reviewer is not bound to any confidentiality agreement, because both papers are public and anybody with knowledge of the field could raise the same concerns. I suppose that sending the previously published paper to the editor with a note wondering why the authors failed to cite their own work, when they use pretty much the same set of data and type of analyses, should be sufficient to raise the interest of the editor.
I don’t see how it could possibly be worth anyone’s time to worry about this. The authors feel it is worth publishing and some journal agrees.
Just do what scientists normally do anyway: gossip about them at the next national meeting.
I have two issues with the reviewer.
1) Did the reviewer tell the editor of the second journal that he reviewed the paper for another journal? If the authors did not make any changes, the reviewer should ask if he can submit the first review. The editor can decide that the authors are not keen on taking feedback or find another reviewer. This has happened to me (I reviewed a paper twice) and the editor appreciated knowing that the authors did not make any revisions after their paper was rejected by the first journal. The editor of the second journal rejected the paper and used my comments to show that the authors were unable to take criticism. This brings me to my second comment.
2) The reviewer said that the paper did not contribute much and the paper was turned down by the journal because the authors refused to discuss and cite their earlier work. Not buying this, the contributions of a paper are subjective. If the reviewer was aware of the earlier work, most readers of the second paper will be aware of the eariler work.
If the new paper suggests that the results are new (as if the authors decided to ignore the contribution of their first paper), the reviewer may want submit his concerns as comments/letter to the editor. if the first paper is well known, the reviewer should expect that his comments will be one of many. If the results are not described as new, the reviewer needs to let it go.
The discussion points out that the other paper was from the “same group”, I assumed that the papers have the same authors but this may not be the case. Having some authors in common would be odd but if the papers are the result of two different groups of researchers and each group worked independently from the other, I do not see why the later one is required to cite the first if they only shared the data.
I work with govt survey data (e.g., BLS, Census) and nonpublic data. I do not cite everyone who has used these data. A colleague and I are doing separate analysis using a set of data that I constructed. I do not expect her to cite my work and I do not feel obligated to cite her paper.
The reviewer should stay out of it and let the members of the group deal with it.
I wonder …. could a situation ever arise where multiple analyses using the same variables within a given data set occur so as to lead to inappropriate interpretations of the results of the later analyses for failure to adjust p values? Sorry, I cannot come up with a specific example at the moment, but I am wondering about the likelihood of such a situation.
Accidentally voted for the last one. Revoted for the one I wanted.
There is another option – find the authors of the earlier analysis on the public data set (assuming its not you), and club together with them to write a letter. Editors generally pay more attention when approached by a consortium of investigators from many institutions, versus an individual who they can write off as disgruntled.
As others have already mentioned, the poll should have a 5th option – comment on PubPeer
I think the presence of option “reviews are confidential” is quite silly… after all, this paper has been published, and anyone can comment, regardless of if they have seen a previous form. Contacting the editor and saying “In a previous version…” would not be ethical. But the paper is in the open, so it’s fair game. Even if the reviewer used his knowledge of the paper’s potential existence to be on the lookout for it, I’d liken this to the inevitable discovery doctrine, whereby the original search may have been unconstitutional/unethical, but the police/reviewer would have likely discovered the paper legally/via pubmed anyway… thus allowing for reporting.
I’m not sure what the ethical concern of the reviewer is. They published work they actually did, they did not plagiarize or fabricate or duplicate. The reviewer thinks that either the findings are not sufficiently novel to merit publication, or the discussion needs improvement. Those are valid concerns and consequently the manuscript was rejected by two journals. A third journal however found the article publishable. This appears to be a matter of judgment about quality and merit, not a matter of ethics.
I take everything back. I overlooked the crucial information: “A previous publication *from the same group* had presented findings on the same topic from the dataset”. Now this is a no-brainer, it’s clearly duplication and should be dealt with as such. The COPE guidelines require retraction if the new analysis doesn’t add anything substantial, and at least a correction if it does add but they omitted to cite their original paper. The question of review confidentiality is irrelevant since both papers are published.
Final comment: When the reviewer encountered the same pattern for the second time, the review could have used some strong language to point out that this was unethical and would not be tolerated. That might have had a deterrence effect.
I wish to second Academic Daylight’s point about the use of strong language in the second review to convey to the authors that their actions were problematic. If doing so would have risked the anonymity of the reviewer, then s/he should have pointed out the problematic nature of these issues to the editor in similar language. Let’s face it, sometimes editors don’t know better (see Wong VS, Callaham ML. Medical journal editors lacked familiarity with scientific publication issues despite training and regular exposure. J Clin Epidemiol. 2012 Mar; 65(3):247-52).
It is now obvious that all submitted manuscripts should be tracked down to compare the initial submission with the final published article. I have encountered major changes ranging from data inflation to adding a control group to a study that lacked them when initially submitted.
First, contact the editors of the first two journals and ask if they mind if you break reviewer confidentiality. s long as they do not mind if you self-identify, then either contact the editor of the third journal, asking to be allowed to write an official commentary, editorial, or letter, that would explain the context of the paper and the relevance of the findings. Not necessarily that you reviewed it before, but that these prior papers exist, and were not cited, and the new paper contradicts or changes the findings of the earlier papers in important ways. Or, if the journal’s web site allows comments, post one.
The problem with not citing appropriate previous analyses in the same data set is that it makes the new paper appear more original than it really is and if he reviewers and editor aren’t aware of the previous publication, then it makes the second paper’s chances of being published better. For instance, suppose I write a paper showing that men consume more zinc in their diets than women. Then my group writes a second paper from the same data showing that younger men consume more zinc than do younger women and older men consume more zinc than older women. If you know about the previous paper, then this is a fairly trivial addition, more like a research letter or a brief communication. But if you don’t know about the previous paper and the authors don’t mention it, then this second paper seems more important. So the point is that if the reviewers/editors don’t know about the first paper, then the contribution of the second paper is exaggerated. The authors benefit from not citing earlier work. And LTEs don’t usually make much of an impression in general and the authors could just reply that they had added more information. So the system is such that actually the authors are probably better off not mentioning their group’s own previous work!
novelty is the trademark here!
Good point! How are editors and editorial boards selected? Any criteria? I was shocked to learn that a very poor academic who could not give a 15 minutes talk about his subject, sat on an editorial board!
I believe that editors selection criteria is worthy of further investigation.
It would be interesting to know what journal editors think a reviewer should do in this situation!
Secrecy never helps justice, period. The reviewers should get all information on paper and its authors. The editor should send the reviews with the names of the reviewers to the authors. Restricting knowledge leads to restricting of the possibilities of argument. Only when everything is known to all parties and the debate is over, the editor can send his decision. You cannot set up a court expecting the judge or a jury to be crooks and make them anonymous. The only, I repeat, the only method of preventing crookery known to the world is TRANSPARENCY.
I fully support the notion of full transparency in science, but I have to wonder whether such a goal is always desirable, particularly in the context of peer review. For example, what happens when a junior member recognizes a fundamental flaw in a study from a crusty, old senior member who is likely to be a reviewer for that junior member’s next grant proposal? Surely, there are other situations in which our biased humanity will trump the objectivity of our reviews. No system is perfect, but I actually prefer an entirely opposite system in which neither, editor, author, or reviewer knows the identities of the parties. I recognize that such a system may be difficult to implement and likely impossible fields with small numbers of researchers all of whom know each others’ work. But, it seems to me that given the biases inherent in all human judgement, the ideal peer review would be one that is completely blind as to who conducted the research and, for that matter, where the research is being published. Stripped of these key elements, what remains is the raw, bare science to be judged only based on its merits.
Ideally I of course agree, but did you consider for instance the fact that a senior member has much more means to find out who is who? Should I post how people are trying to get into my computer?
Don’t know what to say but be your own judge and believe what you know.