Science reported last week that Jens Förster, the former University of Amsterdam social psychologist embroiled in data fabrication controversy, may have stumbled in his defense by muddling the timeline of his disputed studies in public statements.
According to a piece by Frank van Kolfschooten (which is behind a paywall, and to which we linked in Saturday’s Weekend Reads):
The real challenge to Förster’s timeline may lie in e-mails between him and Pieter Verhoeven, his research assistant at UvA from September 2008 to June 2009, who made the correspondence available to Förster’s accuser. In it, the two discuss how to conduct what are evidently the same experiments Förster’s blog declares were completed much earlier in Bremen. For instance, among the stimuli used are three unintelligible audio recordings, which the 2011 paper says were described to the subjects as “Moldavian” poems. In an 18 May 2009 e-mail, Verhoeven comes up with the idea to describe the poem that way, rather than as Malaysian, because the reader of the poem has a German accent.
But in a yet another lengthy open letter to colleagues and friends, Förster insists that he conducted the studies in Germany before coming to the University of Amsterdam. And he hints darkly at the end that those seeking to cast doubt on his research may be doing so for personal gain:
Dear colleagues, dear friends,
As you may have heard, Frank van Kolfschooten (the journalist who published the first NRC article on my “case” some hours after the UvA report was published, and who also wrote the articles that appeared in Science magazine and the Süddeutsche Zeitung) continues investigating my case, citing in his recent Science magazine article from Mai 29 an email conversation between me and a former research assistant. In his new article, the author presents his idea that the studies that I reported had not been done in Germany, but rather in Amsterdam.
Even though printing a conversation with a student in public might be seen as questionable behavior, I am glad that he published this because it illustrates the Kafkaesque situation I am in, in which everything, even standard and viable ideas and decisions are turned against me. As social psychologists we know that if people are convinced about certain facts, this shapes their world views and biases their information search and interpretation (aka as confirmation bias). Eventually, they perceive everything as consistent with their hypotheses. This however can happen unconsciously.
However, the concerns the article might raise can be easily addressed.
First, let me say that all these emails and indications (that are actually based on private investigations by the complainant who had a similar hypothesis in mind) had been examined by the National Ethics Commission (the LOWI). Apparently, all these concerns were unwarranted; this is the reason why they do not even show up in the final evaluation.
If you again wonder about the procedures, I can only tell you: Yes, it is true that the complainant who at the same time was the major expert in statistics during the investigation also interviewed my former research assistants secretly. And yes, it is true that we do not know how s/he asked questions and what questions exactly s/he asked. These conversations are presented out of context.
Second, I do not understand why conducting experiments at UvA logically excludes the possibility that I had done similar ones in Germany. Note that the JEP:G 2009 was submitted in March 2008, and I arrived in Amsterdam summer 2007 – this would have been a rather short time to do all the studies. Note also that in the studies participants had to compare for example “heute” and “tagesschau” – two German news shows that are rather unfamiliar to Dutch participants. Note further that the 2012 SPPS paper contains 390 solutions for a creativity task written in German. Finally, in the Appendix of the 2011 JEP: G paper you find a list with German words. I conclude that the articles that are criticized were not read carefully, and that search biases might have led to wrong conclusions.
Third and most importantly, let me repeat what I expressed in my statement #3 below: I conducted the published studies between 1999 and 2008 in Germany. For outsiders who are not familiar with research in Psychology, it might appear to be strange that I developed stimulus material that allegedly had been used previously.
However, I wanted to conceptually replicate and extend my previously in Germany obtained results with a different (i.e., Dutch) population in a different (i.e., Dutch) language. This requires stimulus material that is suited to test hypotheses with a different population. Just to illustrate this point: Imagine you conducted a certain type of studies with children and want to conduct it later with adults. Of course, you would have to prepare stimulus material for the adults that is different from the one for children. Applying this example to the journalist’s logic, he would wonder why adults would require different stimulus material.
More specifically, the study 1C (from JEP: G 2011) using a “Moldavian” nonsense poem, had been done in Germany. It included a poem for that I changed the vowels and consonants to a fantasy language. The original poem was an old Transylvanian song.
In Amsterdam, I first thought Moldavian would be associated with negative stereotypes (I sensed strong prejudice against East Europeans) and that Malaysian was both more neutral and more believable. Moreover, changing the language would count as yet another conceptual rather than straight replication; something we are looking for. Eventually, however after discussions with the research assistant I decided to take again Moldavian, among others because the poem sounded also to Dutch students more East European than Malaysian, and students considered Moldavians a rather neutral group.
Thus, it is true that I wanted to do similar studies at UvA that included both replications and extensions. I sent basic plans and designs to my research assistant. Obviously the journalist received these emails and files and misinterpreted them. Actually, these files were the beginning of the task for the research assistant: “Let me know what you think, how can this be done, what do you think works best for Dutch students – and if it is impossible for you to figure this out, I can take over”. I wanted his fresh creative “Dutch” input with regard to this paradigm. My experience told me that I cannot simply transport the studies from a German to a Dutch context, rather, some cultural differences (such as food preferences or contents of stereotypes) would apply. I wanted to obtain an unbiased view on materials using the logic from the old basic study set ups to see how they fitted the new environment. Creativity research shows that you block creative thought if you tell too much in advance. In addition, telling a research assistant that using similar paradigms in Germany had already led to many successful studies would have produced tremendous pressure on him. Such pressure could produce unwanted behavior (e.g. experimenter biases) that social psychologists aim to control for. As a matter of fact, such strategy of “not telling too much” is also used in other disciplines and I teach it whenever I teach methods in Social Psychology. However, note that the studies we did at UvA were slightly different (we added modalities to the basic one modality design).
Finally, the SPSS data file from February 2013 contains the original data. Laypeople might not know this but SPSS files get constantly updated. This however does not mean that the original values are changed. Rather, if you translate variable labels from German to English (like “Geschlecht” to “gender”), this file would receive a new time stamp – including the unchanged, original values. In fact, I translated variable labels from German to English in order to make re analyses (for the investigation committee) easier. And please let me use this example to illustrate the unfortunate situation: I wanted and still want to contribute to clarifying the situation. Therefore, I changed the names of the variables from German to English. This or at least the change in the time stamp is now held against me. If I would not have translated the variables, of course, one could have argued that if one fabricates data, I would have used German and not English variable names (to demonstrate that they have been conducted in Germany). In any event – with German or English variable names – I would have found “guilty”. Confirmation Bias in action!
As I said, I gave all these answers to the commissions, and I wonder why the person who passed the material to the journalist did not pass my answers as well – or why the journalist, in case he had the material, did not talk about these simple, unspectacular responses in his article. It is hard for me to believe that this selection happened in the unconscious.
In the end, the lengthy article does not convey any new relevant information. Still, there is no concrete evidence whatsoever of violation of academic integrity. However, this accumulation of negative conclusions, unintended or not, certainly affects my reputation. Note also that some concerns raised in the article were already addressed in my letters and reactions below. I explained in text #3 how I treated outliers and I reported in #1 that an UvA authority figure asked me to dump the questionnaires. Meanwhile I have witnesses for this. Moreover, a former PhD student wrote to me that s/he was asked to dump questionnaires by yet a different person from the department. Ignoring such information is another typical result of a confirmation bias.
In general, I wonder why people publish doubts about my studies that are so obviously unwarranted and that do certainly harm my reputation. Many times misrepresentations are of course lack of expertise to judge the facts (how do we prevent for demand characteristics? how do we prevent for experimenter biases? what do we tell experimenters and why?). However please also note that for some people my case could be profitable.
Regards,
Jens Förster
I really do wonder what this has to do with “Science”. It is a criminal investigation by laypeople, not experts in criminal investigations. Maybe experts, such as lawyers should take over this part, but certainly not non-experts.
A pity that Jens Förster replies to such “contributions”. This is something I do not understand.
I would rather prefer having an academic and scientiic approach to this siutation, which means: more emprical tests of the hypotheses of the relevant papers, more replications, more publications that underwent peer-review, more high-level discussions that deserve to be called a scientific discourse.
Henk, it is easy to understand. As JF states “I wonder why people … do certainly harm my reputation.” It is all about legacy, reputation, and saving both. He is fighting a losing battle. He should admit to errors, and move on. There is only so much of a carcass that sharks can feed on. In a war, as is taking place in science, through publishing, some parties lose, in this case JF, but some parties always gain. His “dark” premonition is not far-fetched.
“I would rather prefer having an academic and scientiic approach to this siutation, which means: more emprical tests of the hypotheses of the relevant papers, more replications, more publications that underwent peer-review, more high-level discussions that deserve to be called a scientific discourse”
This isn’t a controversy over whether Forster’s hypotheses are true. It’s a controversy over whether Forster faked the data in one or more papers.
None of your suggestions would answer the question of whether the papers in question were the result of fraud, or not.
Your suggestions might be of interest for other reasons. But let’s leave it to scientists to decide if they want to follow up on them…
Yes, it is rather interesting that he continues to run his mouth of” in public, considering his legal situation. Most people in similar situations will be told (rightly so) by their lawyers to shut the hell up and let things take their course. If you’re right, trust that the system will make things right and don’t go messing things up by shouting about it. By going public in this way, Forster betrays a shocking level of naivete – all he’s doing is handing his opponents more ammunition, piece-by-public-piece. He must not have very good lawyers (or maybe just chose to ignore them). Compare his behavior with other scientists whose work has been questioned here, and who have chosen not to engage, or have waited to gauge the full situation before issuing a short statement. His current responses do nothing more than add fuel to the fire.
Jens Förster wrote: “I wanted and still want to contribute to clarifying the situation.”
Jens Förster wrote: “Initially, the SPSS data file from February 2013 contains the original data. (…). Therefore, I changed the names of the variables from German to English. This or at least the change in the time stamp is now held against me. (…). Laypeople might not know this but SPSS files get constantly updated.”
I would like to ask Jens Förster to disclose all versions of all these SPSS data files / Excel files on a public website.
See also https://www.knaw.nl/en/topics/open-access-and-digital-preservation/open-access/policy for the policy of the Royal Dutch Academy of Arts and Sciences on open access.
https://www.knaw.nl/nl/thematisch/open-access-en-digitale-duurzaamheid/open-access/beleid (Dutch version).
I wholeheartedly agree. He raises a valid point, but he could just as well have posted the data file alongside his letter, which would have proven this as a faulty argument. Now the situation strongly looks like one where a suspect says he has an alibi, but is refusing to say what that alibi is — either the suspect is bluffing or he is too shortsighted to realize the cards he has. Seeing how prof. Foerster is a smart man, I would hope he realizes the cards he holds and acts upon them — making bluffing a much more likely option.
But hey, my prior might be wrong and I am happy to be proven wrong here.
He should not have changed names in the ORIGINAL DATASET. Many problems in “data fraud” are simply incompetence. Many scientists have NO IDEA about proper curation of data, and his comments about the changes in the ORIGINAL DATASET mean that he should be stopped from doing data analysis, and should hire competent statistical assistance.
he doth protest too much
I see that at least with regards to the dumping of questionnaires, Förster has changed his story. In his first public defence, he wrote:
“The only thing that can be held against me is the dumping of questionnaires (that by the way were older than 5 years and were all coded in the existing data files) because I moved to a much smaller office. […] However, this was suggested by a colleague who knew the Dutch standards with respect to archiving.”
In his new defence, he writes:
“I explained in text #3 how I treated outliers and I reported in #1 that an UvA authority figure asked me to dump the questionnaires.”
Text #3 is the prior defence, so obviously he is misquoting himself.
But more importantly, he is shifting the blame to this unnamed authority figure, who has received a promotion since the last letter from his erstwhile position as an advice-giving colleague.
You mean that he went from “suggested” to “asked” with regard to the input he got from his colleague?
In all fairness, I don’t think we should overinterpret that. If you read the first as “mentioned in passing” and the second as “made an explicit request”, then you would be right that the story has changed in terms of who gave the first initiative to dump materials. But there is the matter of translation, and being a non-native speaker. And even then, both could be right, for instance if the colleague had said something like “Oh Jens, why don’t you just throw those originals away?”. This thing must have happened organically, not after a procedure with checks and balances along the way.
(I don’t mean to defend his actions, but he deserves a normal margin of interpretation in my view, even if his letters were given in writing.)
“You mean that he went from “suggested” to “asked” with regard to the input he got from his colleague?”
No, I mean that:
– The person in question was a colleague and is now authority figure.
– The authority figure *asked* him to throw the questionnaires away, as opposed to merely advising it.
This is a way of saying ‘someone higher up told me to do it’.
We could assume that he doesn’t quite understand the meaning of the words “ask” and “authority”, as you seem to be advocating, but I find this extremely unlikely, given the level of English demonstrated in this defence and his other writings. Moreover, I suspect he deliberately phrased it this way so as create ambiguity.
From https://www.commissielevelt.nl/wp-content/uploads_per_blog/commissielevelt/2013/01/finalreportLevelt1.pdf (page 44):
“Working method and research environment: Tilburg University.
Mr Stapel was appointed full professor at Tilburg University on 1 September 2006. (…). A PhD student from another department was required to tidy up her room. Mr Stapel saw boxes of completed questionnaires and wanted them to be thrown away. The PhD student said she had to keep them for five years. Mr Stapel contradicted her, adding: ‘If you want to commit fraud, you could do it anyway.'”
I would like to ask Jens Förster to disclose the year and month when he moved to this small room. Others are as well invited to disclose this moment.
This is a distraction. Of course Frank van Kolfschooten wants to profit from the Förster’s case. By publishing something largely irrelevant in Science, Frank van Kolfschooten helps himself, but to some extent (probably unwittingly) also Förster.
The focus should be on what by now many people who have looked at the case have concluded (see earlier posts on RetractionWatch regarding this case): “these data are not real”. Förster has now written two open letters, but in neither does he adequately address this issue. Herr Professor Doctor Förster, in your third open letter, please do address it.
Why doesn’t Dr. Forster post the raw data on the internet? If it’s real, unmanipulated data, there is nothing to be afraid of. In fact, if the data are real, publishing the data could help Dr. Forster’s case considerably. Thinking about it, it’s hard to think of any good reason not to disclose the data.
And, incidentally: if Dr. Forster’s story checks out, then there must be additional datasets to the ones published (which apparently were from Germany); namely, the “conceptual replications” for which the student assistant in Amsterdam developed the stimulus materials. I suggest that Dr. Forster publishes all of these datasets on the web asap, so that anyone may verify the claims made in Dr. Forster’s letter.
Sorry for weighing in anonymously. But, Van Kolfschoten sells books on fraud. Isn’t there a real conflict of interest here?
Also, grants are being provided in the Netherlands for “hunting” fraud. Another conflict of interest?
(Disclaimer: I think Forster’s data are crap, but I highly doubt he 1) committed fraud, or 2) if fraud was committed, he did it himself. Why does the UvA not investigate ALL his papers? This would be best for the question of fraud and for Forster himself).
Please provide more information about the grants for hunting fraud, web-sites included. Also, any advice on what authorities to contact about fraud or misconduct by Dutch scientists.
Frank van Kolfschooten ( http://frankvankolfschooten.nl/wordpress/ ) is a writer / journalist who has published books on the topic of scientific misconduct of Dutch scientists.
His most recent book is ‘Ontspoorde wetenschap’ ( http://www.uitgeverijdekring.nl/boek/ontspoorde-wetenschap ). Journalists, as well as scientists, can apply for grants. Frank van Kolfschooten got grants from ‘Fonds Bijzondere Journalistieke Projecten’ and from ‘Stichting Democratie en Media’, which have been used to write this book. It was published in 2012 (only in Dutch).
These are the more general grants, so I would have to point to individuals, which I prefer not to do.
Only if there is a genuine conflict of interest (e.g., complainer receives grants, leaks to press, etc.) should this be looked at. It’s a question, not an accusation. But this process needs to be done carefully, just as carefully as the Forster case, because if that would not be true, then people are also falsely implicated.
Van Kolfschoten’s conflict of interest is clear though.
I do not really see the conflict of interest: FvK did not initiate this, he did not identify or analyze the regularities in the results in JF’s papers, he did bring this case before the (academic) authorities, finally, JF does not answer to FvK. FvK is a journalist with an interest in scientific fraud, and as such – naturally – he is pursuing it.
If web sites like Retraction Watch or science journalist like Kolfschoten do a professional, open and transparent job in investigating and exposing scientific misconduct, they should be allowed to benefit from that endeavor. The same way that psychologists like Förster should be allowed to benefit from doing professional and pioneering research, for example by receiving a well payed Humboldt Fellowship. This is better explored in the open than by some big shots holding secret conferences behind close doors. Of course this most not become a witch hunt, everybody must keep their mind open. I wonder what the people at the Humboldt Foundation are doing now; surely they also read all this stuff.
Scientists need to follow all kind of rules before they can make public statements about their findings / research results (etc.). The same is the case for journalists.
Any journalist must work according to the principle of “hear the other side too (audi alteram partem)”. I don’t have access to the recent publication of Frank van Kolfschooten in Science. I would like to ask others to clarify what kind of information is given in regard to the opinion of Jens Förster about the various statements in this story in Science.
You can find an unformatted version of the text here: http://pastebin.com/7WUNJu5b
There’s an article written in a newspaper before all the evidence is clear. When there is a conflict of interest, and people benefit from fraud detected, it’s not unlikely that available evidence is not weighted carefully.
But, again, this can all be remedied if the university investigates all his paper (but, again, the process is rather dubious. Siminsohn’s trajectory in previous cases was MUCH more careful). Compare the Smeesters and Sanna cases, compare to this.
1) Invariably, newpapers articles are written before all the evidence is clear. What do you expect journalists to do? 2) Invariably, interested parties present the facts or “their facts” in a manner that serves their case. What do you expect those involved (including JF) to do? (one would hope that the LOWI, as an independent disinterested party, jugded this case impartially and fairly; the report is online (in translation)). Finally, you make mention of “people benefit from fraud detected …” That is quite vague (as is the statement “it’s not unlikely …. not weighted carefully”). Who benefits from this fraud? How is all this a benefit to anyone?
I think that clearly the journals is profiting from this! At least his name gets known by more an more people……
It is amazing that Science publishes articles with conclusions that are based on such dubious evidence. Next time, phone conversations will be tapped to undermine scientists’ allegations.
JATdS wrote: “Also, any advice on what authorities to contact about fraud or misconduct by Dutch scientists.”
You will first need to find out the affiliation of these Dutch scientists, in most cases one of the Dutch universities.
Afterwards, you need to check their Academic Integrity Complaints Procedure. See http://www.uu.nl/university/utrecht/EN/profile/profileandmissionstatement/Pages/Academic-Integrity-.aspx for an example how to handle in case you want to file a complaint against a scientist who is affiliated to Utrecht University.
All Dutch universities have agreed with each other to follow the same kind of procedure (though there might be subtle differences). All Dutch universities have an Ombudsman who can also be approached. All complaints can be send by e-mail. All Dutch universities have an English version of (parts of) their policy on Academic Integrity on their website. See also http://www.wageningenur.nl/nl/Over-Wageningen-UR/Integriteit/Wetenschappelijke-integriteit.htm
Before filing a complaint, please read and re-read “The Netherlands Code of Conduct for Scientific Practice” ( http://www.vsnu.nl/files/documenten/Domeinen/Onderzoek/The_Netherlands_Code_of_Conduct_for_Scientific_Practice_2012.pdf ) several times, as well the “National Model Complaints Procedure Academic Integrity of the joint Dutch universities”, see http://www.uu.nl/SiteCollectionDocuments/Corp_UU%20en%20Nieuws/Klachtenregeling%20WI%201%20september%202012_EN%20def.pdf for the version of Utrecht University.
Both documents are complementary to each other. The Code describes desirable conduct, the other document describes undesirable conduct and how to deal with with undesirable conduct.
One paragraph: “Finally, the SPSS data file from February 2013 contains the original data. Laypeople might not know this but SPSS files get constantly updated.”
Perhaps this is a translation thing, but on its face, it is totally false. Files do NOT get “constantly updated”. One of the GREAT SINS in data analysis is to modify the original data file. You modify analysis datasets, but you never never ever ever touch the original data. You must NEVER change that.
“Rather, if you translate variable labels from German to English (like “Geschlecht” to “gender”), this file would receive a new time stamp – including the unchanged, original values. In fact, I translated variable labels from German to English in order to make re analyses (for the investigation committee) easier. And please let me use this example to illustrate the unfortunate situation: I wanted and still want to contribute to clarifying the situation. Therefore, I changed the names of the variables from German to English. This or at least the change in the time stamp is now held against me. If I would not have translated the variables, of course, one could have argued that if one fabricates data, I would have used German and not English variable names (to demonstrate that they have been conducted in Germany). In any event – with German or English variable names – I would have found “guilty”. Confirmation Bias in action!”
The original data file SHOULD NEVER CHANGE.
Of course you are correct. I thought the same thing when I read Förster’s response. Would he now be in possession of the data set with an old and with the new timestamp, it would be quite convincing that at least this part of his story is true. But he apparently changed the original data file. Of course, that was not a wise thing to do – then again, he also did not keep the original *data*, so at least he’s being consistent.
Says who? Are this the standard guidelines in Psychology?
If so: I agree!
If not: The post should address Psychology in general. I constanly see posts here that apply their own ideas/rules/guidelines to this very speciific case and do not reflect on whether the same ideas (standards, rules, etc.) are applied in Psychology or in Research in the Netherlands/Germany. If they would have been valid for Jens Forster as well at the time he published the results, he should be critizised for this.
But not because some non-applicable ideas (as good and valid as they might be) are applied in retrospect.
I think that if anything, the Förster case and the many other recent cases in (social) psychology show that there should *not* be separate ways of doing things in the field of psychology. So, in my opinion, the question should be: are these the standard guidelines for doing empirical research?
Look through Retractionwatch and count the cases in other fields. The difference is that in psychology, one case causes four entries and hundreds of comments. Perhaps in psychology, such things are rarer and therefore more newsworthy than in other disciplines, where they are understood as side effects of “normal science”.
Totally agree. Good guidelines, but they were not applied in the past. You cannot use that as evidence for fraud if the norm (however poor the norm) was not like this.
That said, like other commenters indicated: JF should say, yes, we can do science better, my data wasn’t great, let’s do this stuff better.
I am just wondering, what this recent “investigation” is all about.
The data show (as far as I understood) a strange pattern of linearity.
Now all the other information try to show what? That the studies had been conducted in Amsterdam?
Let’s assume that Jens Forster would have had provided the data and questionnaires, would this change the situation?
No, there would still be the strange pattern of linear relationship
In NRC handelsblad (the first article) van Kolfschooten wrote “UvA-hoogleraar manipuleerde data van onderzoek”, which means “UVA Professor manipulated research data” although this was not even the result of any investiagtion (also not the LOWI). The result was that the data hadbeen manipulated – by whom it is not clear and this has explicitly stated by the LOWI.
I agree. Science has published a new piece on the case that was written by a different journalist.
http://news.sciencemag.org/people-events/2014/06/embattled-psychologist-addresses-new-challenge
Freddie, thanks for the link. Martin Enserink wrote: “Förster, who recently resigned from UvA, did not respond to multiple requests for comment on the Science story.”
Jens Förster wrote (29 May 2014): “I wonder why the person who passed the material to the journalist did not pass my answers as well – or why the journalist, in case he had the material, did not talk about these simple, unspectacular responses in his article.”
Jens Förster wrote (29 May 2014): “I wonder why people publish doubts about my studies that are so obviously unwarranted and that do certainly harm my reputation.”
“Any journalist must work according to the principle of “hear the other side too (audi alteram partem)” (auto quote).
So Frank van Kolfschooten has published his articles in line with this code. Can anyone explain why Jens Förster did not respond to these requests so he could discuss with Frank van Kolfschooten all his concerns before the articles were published in Science?
Jens Förster wrote (11 May 2014): “Note that the acknowledgment sections in the papers are far from complete (…)”
Jens Förster wrote (29 May 2014): “Note further that the 2012 SPPS paper contains 390 solutions for a creativity task written in German.”
Jens Förster wrote (29 May 2014): “I conclude that the articles that are criticized were not read carefully, and that search biases might have led to wrong conclusions.
Please show me where it is written in the 2012 SPPS paper that these participants were working with tasks written in the German language. Please not that I don’t have access to the contents of 2011 paper (“Pretests as reported in Förster 2011”).
Please show me those incomplete parts of the acknowledgments in the 2012 SPPS paper (ignore the default statements at the end about CoI and funding).
As deliberate data manipulation has already been proven through various statistical analyses (see the accuser’s report, see the LOWI report, see the datacoloda website, see Neuroskeptic’s website), the other circumstantial evidence is only interesting for people who do not understand the statistical argument.
Here is another yet another piece of ‘other circumstantial’ evidence. Forster himself says: “Let me repeat that I never manipulated data. However, I can also not exclude the possibility that the data has been manipulated by someone involved in the data collection or data processing.”
I would hope that Forster has had sufficient statistical training to understand that if he did not manipulate the data himself, that then surely someone else must have manipulated the data. And if Forster did not manipulate himself, then why isn’t he interested in finding the person who did?
Forster should have no problem at all to identify this person, because it must be someone who had access to the data of ALL experiments that have been published, in order to manipulate them all in the same way. Moreover, he or she must also have been in the position to remove ALL copies of ALL original and ALL other pre-processed versions of the published data files.
In addition, given that the published experiments are selected from a set of hundreds (?) of experiments: where are all the copies of orginal, pre-processed data files of all these other experiments? There must be many assistants who helped Forster and who did not experience unfortunate hard disk crashes.
Forster should have no problem to come up with original versions of most of the published and unpublished experiments. This would be an easy way to defend himself if he would be an innocent victim of an over-enthusiastic data-assistant.
It is interesting, how laypeople try to use legal procedures: One has to prove that someone is guilty (here: that Forster himself has manipulated data) and not to prove that someone is not guilty. However, this is what has been claimed here and in other posts.
The a priori verdict is guilty (without any evidence) and now some vague assumptions and opinions seem to replace evidenve.
I am not convinced.
Denny Borsboom, Han van der Maas & Eric-Jan Wagenmakers, Department of Psychological Methods, UvA: “Dr. Förster suggested that our department would be involved in a witch-hunt. This is incorrect.”
See http://osc.centerforopenscience.org/2014/05/29/forster-case/ for more details.
Interesting. Thanks for this. But the possibility of QRPs is quite open, according to this story.
So why is fraud being pushed as the main option? It seems *likely* that at least a very strong and very questionable QRP tactic was used, but is that fraud? Fine line, but given that the norms are changing, in the case of very bad QRPs, I would give JF another chance (not in the case of fraud). But he’s so much on the defensive. Coming clean about QRPs seems more commendable.
Though, given the current atmosphere, co-created by many, it is tough to come clean on QRPs..
What “current atmosphere, co-created by many”, actually? There are many and varied comments here, pro and contra JF, the details of the case against him, and his responses. Do you perceive an anti-JF atmosphere? If so: does your accusation of a very strong and very questionable QRP tatic contribute to it, given that JF strongly denied (in private session with the LOWI, where there was no questions of your perceived “current atmosphere, co-created by many”) that he indulged in any QRPs at all? You seem to be suggesting that JF lied, but that the so-called atmosphere here at RW is somehow extenuating. I do not think that that makes much sense.
There is one major problem with this theory: in the LOWI report, the statistician consulted by the investigative commitee was specifically asked whether the abnormalities could be explained by QRP. The statistician answered that s/he could not.
Further, Förster himself has (presumably unknowingly) already admitted to QRP in his open letters…
I think that the atmosphere is a tense one. I just stumbled across another case in Psychology, in which a failed replication (e.g., http://www.psychol.cam.ac.uk/directory/[email protected]) lead iho to some questionable comments (e.g., https://twitter.com/JelteWicherts/status/468612079721521154)
And of course, the case here and the Forster case are completely different, the judgmental atmosphere isn’t.
The debate that was sparked by the recent Special Issue in “Social Psychology” is an awful, AWFUL example of the culture of (psychological) science. Here’s a bit more if you need further evidence:
https://twitter.com/DanTGilbert/status/470199929626193921
It has little to do with the Förster case (at least one hopes) specifically, as the debate mostly regards whether some failed replications are conclusive or not. I find it therefore a bit unfair (to both Förster and the “replication discussants”) to compare these two debates only because there seems to be a heated argument.
Again, Uri Simonsohn has written an excellent comment on the whole thing, I’d recommend reading this one over the debate that spawned on Twitter and various blogs, forums, etc.
http://datacolada.org/2014/06/04/23-ceiling-effects-and-replications/
I meant to post a link to a whole discussion thread on Twitter, not just the one by Gilbert, but it automatically changes. Click on “3:49 PM – 24 May 2014” just below the Tweet to read the whole thing.
I totally agree that these are different cases – regarding the content.
But I am really surprised of the tone of the discussion from some people in Psychology (I assume that most of them are Psychologists).
At least (this might be a prejudice now), they should know how to communicate.
And sometimes (!) the tone – on both sides (against/in favor of Forster or Schnall) – is more like “a throw you in front of a bus – enjoy”.
This suprises me – in both cases.
And I am not convinced that this is just the “heat of the moment”.
“In fact, I translated variable labels from German to English in order to make re analyses (for the investigation committee) easier.”
Let me get this straight. You are under investigation. Your (scientific) veracity has been called into question. And when the investigative committee asks for your data set, you edit and *overwrite* the original file before handing it in.
And all this, apparently, in an effort to assist the committee, who cannot be trusted to either know German or use a dictionary? Give me a break.
Statistical Observer wrote: “The original data file SHOULD NEVER CHANGE.” Henk Koost wrote: “Says who? Are this the standard guidelines in Psychology?”
Henk, guidelines on keeping a close record of original data files exist for already 100s of years. The original handwritten labels of specimens of animals and plants in collections in musea all over the world must always stay attached to the specimen. One can make a new label (eg with new information), but that does not mean that one is allowed to throw away the old label. No way.
I fail to understand why you have the opinion that such a general guideline might not apply for primary data collected during psychological research conducted in any European country.
I am not saying that they might or do not apply to Psychology. However, just because they are valid in some/most disciplines, it does not show that
a) they have been valid in Psychology as well (or are valid)
and
b) that they have been thoroughly applied in (Social)-Psychology as a standard procedure.
If a) and b) are true, then Forster should explain – in detail – why he did not follow them.
If a ore b are not true, one needs to focus on the entire field of Psychology/Social-Psychology.
Claims and the citaition of some general procedures – in my eyes – do not prove anyhting.
Dear Henk, within science as a whole we should expect general principles such as retaining original data, to be inherited by default from a discipline into its subdisciplines, without having to be re-enumerated within each subdiscipline. If this were not the case, then I could always wriggle out of any convention by declaring myself to be my own subdiscipline, and therefore exempt from any general standards.
Thus surely social psychology ought to, by default, inherit the general principles of experimental science?
I agree, Darrel.
What I meant, however, is a diagnostic problem. If anyone or most people in a certain discipline do X (it might be wrong or right), it does not say much about a person A from this field who does the same. Or at least the base rate of behavior X needs to be taken into account.
In my eyes, this basic principle is repeatedley violated here and throughout the whole process.
But regardless of the corporate culture in soc. psych. (which, I think, has undergone a major revision w.r.t QRPs following the Stapel case – we can thank Stapel for that at least!), the fact remains that JF presents results that none of his colleagues in the field presented (near perfect linearity, unprecedented large effect sizes, low variation in means). So that suggests that the corporate culture – while an interesting issue in its own right – is ultimately of minor interest in explaining JF’s results. If most in the field were doing X (be it right or wrong), why did not most people in the field produce such immaculate results as JF’s? (I note that the whistle blower’s report contained results of others in the field that looked nothing like JF’s).
Could you elaborate more on your theories of this “corporate culture”, perhaps provide some cases to support the theory and even literature? Why d you feel that such a culture is unique to soc. psych.? More detail is rquired in order to try and draw parallels, if any, with other fields of science.
I was assuming that Henk Koost was referring to Soc. Psych. (i.e., “the certain discipline”), but perhaps it applies to psychology in general, or science in general? That would not detract from the point I was trying to make.
Your refer to “my theories” and “my theory”, but I lauched no theories in my brief response. I was trying to suggest that practices (good or bad) in “a certain discipline” may certainly be of interest, but are really of limited interest w.r.t JF’s results: assuming that JF and his colleagues in the field all embrace such practices (JF produces results which his colleagues did not). Btw: what practices exactly is immaterial to my point. I was responding to Henk Koost reply to Prof D. Francis. I hope this clarifies the point I was trying to make.
Unfortunately, it is way off the explanation I was hoping to hear. You used the word “corporate”. Corporate means belonging to a corporation (sensu stricto). Dd you misuse the word, or are you unsure of the corporate link?
Thanks. I was using the term figuratively. Taken literally, it is an misuse of the term. I had not anticipated a literal interpretation. It is unfortunate as it is a distraction from the point I was trying to make. To unscore my meaning: “In psychological experimentation, one should correct one’s alpha to arrive at a family-wise alpha if one conducts many tests in one and the same experiment”. “In psychological experimentation, one should conduct power analyses before conducting statistical tests, otherwise the failure to reject the null-hypothesis is hard to interpret”. Such practices are part of the corporate culture. Figuratively I call such practices part of the corporate culture, where I merely mean to say: part of the experimental research culture. If this is “way off the explanation” you were hopeing to hear, I am sorry to disappoint you. What explanation were you hoping for?
While I agree with above posts that this email exchange is of limited importance here, it is nevertheless interesting.
Forster claims that he had used Moldavian as the “language” of the nonsense poem in Germany. Then he comes to Amsterdam, tells his assistant Verhoeven that he wants to do a study and proposes Malaysian as the language. And Verhoeven comes back with the idea of Moldovian? Quote according to science: “I think it’ll work when we use a Eastern European language, like telling that it’s a Moldovian poem.” Just by chance Verhoeven picked the same obscure language, or what? Ok, you could still argue that Forster told him about the German study. However, Verhoeven apparently didn’t write “I think we should just stick with what you used in Germany, Moldovian or what was it?”. That’s what anyone aware of previous studies would write IMHO. Verhoeven’s answer sounds very much like Moldovian was his original idea. How are the chances that someone comes up with the same idea that Forster had a few years before in Germany? And then two years later Forster actually publishes a study with the use of a Moldavian nonsense poem. And the primary data (questionnaires) are all gone. This really doesn’t help support Forster’s credibility.
Despite that, the question of the statistical anomalies in the data has not been addressed appropriately by Forster and that remains the big issue.
BTW, are there ethics commities involved in psychological studies? There must be some records in Bremen indicating that the studies did take place and what was done…..
Interestingly enough, Jens Förster himself in 2005 signed the document that clarifies in which cases social psychologist have to report their research projects to the ethics committee. I do not think that his own study was affected:
https://www.psych.uni-goettingen.de/de/ethics/Richtlinien_DGPs.pdf
Good point! They should have records on the studies – if they really have been conducted during that time.
The “Richtlinien” is about to document for DFG, if you do any problematic research, for example with children, or if you cheat the subjects in order to obtain results. Its not a “Richtline” to document when or where a study was conducted.
But: I know from studies for example in social psychology in Mannheim/Germany that they follow the general guidelines from DFG if they get funding and every subject has to sign a declaration of consent to document that they took part in studies voluntarily, see here:
http://www.sowi.uni-mannheim.de/lssozpsych/studien/ic_stahlberg_Online.pdf (it’s for online studies, but I know they have this for laboratory studies, too)
Does someone know if Förster received funds from DFG while conducting his problematic studies? Then he has to document any of the declarations of consent from subjects to DFG.
The SPPS 2012 paper ( http://spp.sagepub.com/content/3/1/108 ) states: “Funding. The author(s) received no financial support for the research, authorship, and/or publication of this article.”
I don’t have access to the full text of both other papers with questionable data (Förster 2009, JEPG 138: 88-111 and Förster 2011, JEPG 140: 364-389). Quite a few other (older) papers (co-)authored by Jens Forster do list a DFG grant (various grants in different papers).
How about all regulations listed in
http://www.dfg.de/download/pdf/dfg_im_profil/reden_stellungnahmen/download/empfehlung_wiss_praxis_1310.pdf ?
Staff and students at Jacobs University (Bremen, the former workplace of Jens Förster) must also work according to these rules of DFG.
False alarm. Finally, I searched myself the two other documents.
But could not find any hints linking this papers to the DFG.
In Germany IRBs or similar committees are relatively rare for psychology/social science departments. So I assume these records do not exist. However, other records should exist, e.g. records of money or credit points given to participants as incentives.
But that would only confirm that some studies were conducted. It says nothing about the study designs, which is what an ethics board would have records about.
We already have a strong indication that Förster really did conduct studies in Germany: one of the research assistants who worked for Förster in Germany confirmed in a prior article on this website that studies were run there.
The most recent Van Kolfschoten article therefore does not suggest to me that the data is completely fabricated, but it adds evidence that the study design was misreported, strengthening the case for misconduct.
The chapter
“Finally, the SPSS data file from February 2013 contains the original data. Laypeople might not know this but SPSS files get constantly updated. This however does not mean that the original values are changed. Rather, if you translate variable labels from German to English (like “Geschlecht” to “gender”), this file would receive a new time stamp – including the unchanged, original values. In fact, I translated variable labels from German to English in order to make re analyses (for the investigation committee) easier. And please let me use this example to illustrate the unfortunate situation: I wanted and still want to contribute to clarifying the situation. Therefore, I changed the names of the variables from German to English. This or at least the change in the time stamp is now held against me. If I would not have translated the variables, of course, one could have argued that if one fabricates data, I would have used German and not English variable names (to demonstrate that they have been conducted in Germany). In any event – with German or English variable names – I would have found “guilty”. Confirmation Bias in action!”
suggests between the lines that Dr Förster edited the file “to make reanalyses easier” but probably in a way that he is not disclosing here and at the same time translated the variable names – which seems both unnsecassary and stupid as in an ongoing investigation a primary rule is not to tamper with evidence – so as to hide his track. He dicks his hole deeper with every explanation.
Please be aware that German is a mandatory subject for at least a few years for anyone attending a Dutch high school (in particular for the type of high school which gives access to any of the Dutch universities), and please be aware that both languages are quite similar to each other.
Please also be aware that quite a few (most? / almost all?) Dutch scientists can understand German texts with a firm relationship with their field of research just as good as they had been written in English.
More about Jens Förster:
https://plus.google.com/101046916407340625977/posts/1pKwsaLKA1G