Last week, we wrote about the story of Paul Lodder, a graduate student at the University of Amsterdam who had been trying without success to replicate the findings of a 2020 paper in Scientific Reports by Rubén Herzog, of the Universidad de Valparaíso in Chile. The paper would end up retracted. At the time, Lodder had not had a chance to respond to our questions about the case. We’re pleased to share his comments as a guest post.
I’ve had a big passion for research into the therapeutic potential of psychedelics ever since I had started my undergraduate in biomedical sciences at Amsterdam University College. I am currently a MSc Artificial Intelligence student and about a year ago, in preparation for a computational neuroscience course, I wanted to expand on the model used by Rubén.
I sent him an e-mail explaining the situation, and requesting some parameters that weren’t detailed in the paper so that I could start running the simulations myself. Rubén responded very quickly and was immediately very helpful with getting me started with running the simulations.
Now that I was able to run the model properly, I wanted to start off with being able to reproduce the paper’s analysis results before looking into expanding. Using the methodology described in the paper, I re-implemented the steps needed to compute the entropy. And indeed, this is where I got some different results as presented in the paper.
I shared my findings with Rubén by writing up a literate programming report, which allowed me to present my findings in a reproducible manner by including the code that leads to any of the results discussed.
Rubén again responded helpfully by suggesting some directions in which to look for explanations of the observed discrepancy. After I had explored each of the options and communicated the results with Rubén, he sent me his original code. I produced some simulation data, ran Rubén’s code on that data and was able to reproduce the paper results exactly.
I then ran my own code and found the same as before, which meant it was not in agreement with the results in the paper nor that Rubén’s code has just produced. After having a closer look at Rubén’s code I found the typo, which, when corrected, lead to the exact same results as I had been getting.
Because I was able to run his code and mine on the same simulated data, it was very clear that the typo caused the results from the paper. I shared my results with Rubén in a written report and he quickly got back to me. He acknowledged that this was indeed an error and thanked me for trying to reproduce his results. He also invited me as a co-author for the resubmission, by which I am truly honored.
I would like to highlight that it is only because Rubén had been so helpful throughout the process that I was ultimately able to spot the typo: He helped me get started with the simulations, thought along with me when I communicated my findings, and finally even shared the code with me to see if I could find an explanation for the discrepancy. Rubén’s handling of the situation has been a display of extreme academic integrity from beginning to end.
In a more general sense, however, I’m disappointed in how little emphasis there seems to be in academia (and by journals) on the reproducibility of computational results. Unlike in any other discipline, papers could be accompanied by a script with code that not only analyzes, but even produces the very results on which the paper is based.
Of course it need not be as extreme as a single script that reproduces everything from scratch, but some documented code would already go a long way. Not only would this lead to greater reproducibility, but it would accelerate scientific research too, as it eases the process of building on what others have previously done.
Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].
Integrity is heroism
This remindes me of my long-planned addition to my lab’s standard operating procedure: to have all code for the statistical analyses of a paper written by two independent coders. If they arrive at the same results, then chances are high that the results are valid and do not depend on a coding glitch. If they arrive at different results, one (or both) of the coding approaches was (were) invalid. In an ideal world, scientists would have the time and team size to be able to be this thorough and meticulous when preparing their publications. But in the messiness of real academic life with its limited resources (time being at the top of the limitation list), such a goal may be difficult to achieve. Thus I guess for the time being we need to rely on the eagle-eyed Lodders among our colleagues to spot our missteps and have Herzogian integrity in our response to detecetd flaws.
Great policy!
My MSc advisor, a theoretical and computational biologist who was trained as a physicist, would ask us students to check his code and vice versa. I’ve taken this science over ego philosophy with my own students and hope they carry this torch.
What an incredibly honorable & appropriate outcome on the part of both Rubén Herzog and Paul Lodder. Lodder’s efforts to understand the error is notable. Offering Lodder the chance to co-author shows real class on Herzog’s part. Reading how Herzog handled the situation suggests he’s a very good mentor. 2 honorable & classy individuals from 2 parts of the globe
Thanks Paul and Rubén. Reproducibility and accountability. This is how we are supposed to soldier on in practical research.
This is an excellent example of cooperation and integrity. My colleagues and I had a completely different experience when we investigated Fenton, J. J., A. F. Jerant, K. D. Bertakis, and P. Franks. 2012. “The Cost of Satisfaction: A National Study of Patient Satisfaction, Health Care Utilization, Expenditures, and Mortality.” Archives of Internal Medicine 172 (5): 405–11. The authors would not share their code so we had to recreate it from scratch to reanalyze the data. Xu, X., Buta, E., Anhang Price, R., Elliott. M. N., Hays, R. D., & Cleary, P. D. (2014). Methodological considerations when studying the association between patient-reported care experiences and mortality. Health Services Research. 50(4), 1146-61.
‘extreme’??? I would say that they behaved well, as would be expected. I did not murder anyone, should I be lauded for extreme pacifism?
I think you under estimate the dedication of these two Scientists. They had to do a lot of work to correct the problem- it was not a passive undertaking.
A better analogy would be, a civilian finding a problem with the DNA results for a murder conviction and informing the police and justice departments, at which point these organisation exonerate the accused.
Sit down, if you follow RW at all, it should be very clear how often something like this happens. More often, you see authors getting defensive and doubling down on errors rather than actually working through the problem with integrity and science at the fore.
*** EXTREMELY BRILLIANT RESPONSE *** I was struggling to find an appropriate response!
Someone should build a statue to you for your expected behavior ! And everyone that behaves well and in *gasp!) Expected ways normal to the specific environment !
Patina and pigeon droppings aside, your response obviously resonated with me, and I find it applies to both behavior and achievements.
Once with friends one evening we were at JPL near the holidays for Galileo’s JOI – The Jupiter Orbital Insertion burn.
Along with the good comedy afforded by JOI to The World hymn including references to the SLOW data rate from the partially unfurled primary communication antenna, there was a live video display of the Doppler shift of the Galileo radio signal on a wall monitor.
The spacecraft path was confirmed as Nominal by many means, but the fastest indication of successfully being captured by Jupiter’s gravity was a simple “Hockey Stick” plot, down and to the left showing the speed of Galileo using the Doppler Shift of the carrier signal. Just a single scalar plot.
Every time we looked at the plot it was “Right Down The Pipe” dead on the Nominal prediction and perfectly between the limits plotted for reference.
It wasn’t exceptional. My wife and I finally deeply understood that it was NOMINAL and that there wasn’t anything better than what had been planned for and engineered to achieve. Just NOMINAL. Not “better than expected”, it was the expected outcome.
My wife looked at me the moment gravitational capture was confirmed and people were celebrating, then pointed to the nominal plot on screen and said “I guess that’s what they get paid for !”
We’ve used that endearment whenever someone’s extreme efforts go NOMINALLY, as planned and as expected.
My point is that behavior and achievement can be expected and Nominal, and that in no way makes it mundane !
But I sure seems odd to say expected behavior is “extreme”. My experience with humans is better than this implies.
Reading this makes me realize how uneducated I am.
You have no idea, clearly, how rampant is academic theft. It ruins lives and careers, making one researcher a hero and another a zero.
Lodder could have refused to give Herzog credit due, likewise, Herzog could have refused to assist Lodder.
Both men did a highly honorable thing.
When you start to work on your Masters thesis or doctoral dissertation, you’ll understand how incredibly stupid you sound, @JG.
Oops
Plot was Down and to the RIGHT.
Correction. No need for retraction yet!