Mean streets: Expert on lying accuses planning association of ethical lapses

A U.K. urban planner and self-styled expert on “truth and lying” has launched a forceful attack on the ethics of a key trade association, accusing it of refusing to promote his work for fear that the findings might be damaging to the profession.

And what, you’re asking, does this have to do with retractions? Trust us. This story’s harder to follow than a New Jersey left turn ramp — but we think you’ll enjoy it.

As a road map, here are a few key players in the drama:

• Bent Flyvbjerg, a planning researcher with a sidelight in “phronetic social science” — “an approach to the study of social phenomena based on a contemporary interpretation of the classical Greek concept phronesis, variously translated as practical judgment, practical wisdom, common sense, or prudence” — at University of Oxford
• The Journal of the American Planning Association (JAPA), which in 2002 — and 2005, but more on that later — published one of Flyvbjerg’s articles in which he and his colleagues blamed dishonesty on the fact that major public works projects tend to cost more than planners estimate.
• The American Planning Association (APA), publisher of JAPA, which, Flyvbjerg says, agreed to aggressively promote the 2002 study then shied away from publicity when it began to fear that doing so might put its membership and the profession in ill repute — what he calls “uncomfortable knowledge” that the group tried, in Kübler-Ross fashion to deny, dismiss, divert and displace.
• Cities, a competitor journal to JAPA, which is publishing an editorial by Flyvbjerg laying out the tale in excruciating detail, including anonymous quotations from an APA media rep who allegedly told him that the group “does not want me to do anything in terms of promoting, whether as an exclusive story or on a regional basis, the results of your infrastructure cost underestimation study … I’m sorry I can’t be of further help.”

As it happens, the 2002 paper did garner substantial attention in the mainstream press, including the New York Times and the Boston Globe, which quoted him saying:

“The key policy implication for this consequential and highly expensive field of public policy is that those legislators, administrators, bankers, media representatives, and members of the public who value honest numbers should not trust the cost estimates,” Flyvbjerg writes, adding in an interview, “We should expect the status quo if things don’t change. But they should.”

Okay, now that that’s all clear, here’s what Flyvbjerg has to say in his “study of moral hypocrisy” in Cities:

 APA was found [by himself] to actively suppress publicity of malpractice concerns and bad planning in order to sustain a boosterish image of planning. In the process, APA appeared to disregard and violate APA’s own Code of Ethics. APA justified its actions with a need to protect APA members’ interests, seen in preventing planning and planners from being presented in public in a bad light.

We’ll remain agnostic on this part. But we do agree with Flyvbjerg’s next point — because it’s one we’ve made repeatedly on this blog and elsewhere:

The current article argues that it is in members’ interest to have malpractice critiqued and reduced, and that this happens by exposing malpractice, not by denying or diverting attention from it as APA did in this case.

But we do wonder why Cities would publish the editorial, especially since the events it deals with are now a decade in the past. Flyvbjerg tells us:

Publication of the research was delayed in order to protect informants. Many of these have now moved on to other positions. Publication was also delayed in order to emphasize that the study is about matters of principle – malpractice and professional ethics in planning – and not about individuals and individual actions. There is no indication there is less reason today to be concerned over the ethical issues raised in the paper than ten years ago, but quite the opposite.

Fair enough. But it’s also a touch ironic — and, perhaps a bit more than that — for Flyvbjerg to make this case, given this:

Statement of Retraction

Upon a thorough investigation, it has been determined that the paper titled “Inaccuracy in Traffic Forecasts” by B. Flyvbjerg, M. K. S. Holm and S. L. Buhl published in Transport Reviews, 2006, volume 26, pages 1–24 is substantially similar to that of an already published article titled “How (In)accurate are Demand Forecasts in PublicWorks Projects? The Case of Transportation.” by Flyvbjerg, B., M. K. S. Holm and S. L. Buhl, Journal of the American Planning Association, 2005, volume 71, pages 131–146.

As a result, the article published in Transport Reviews has been retracted and should not be cited in the electronic or print version of the journal.

That retracted paper has been cited 31 times, according to Thomson Scientific’s Web of Knowledge. And we found that Flyvjberg may have double-published the 2002 JAPA paper he wanted better publicity for in, of all places, JAPA, in 2005. As this site notes, the two articles are quite similar — with at least one obvious spot of self-plagiarism and the distinct whiff of salami slicing:

Publication in April 2005 of a new study by Bent Flyvbjerg, Mette K. Skamris Holm and Soren L. Buhl brought back memories of an earlier work by the same authors.

The 2002 paper by Flyvbjerg et al., Underestimating Costs in Public Works Projects: Error or Lie? The 2005 paper, How (In)accurate Are Demand Forecasts in Public Works Projects? The Case of Transportation, was also published by the APA Journal. Flyvbjerg (pronounced much like FLEW-byair; rhymes with “Pierre”) and Buhl are professors at Aalborg University, Denmark. Skamris Holm is a planner with Aalborg Municipality.

The 2002 paper analyzed construction cost estimates for large public works projects using statistical methods. Flyvbjerg and his collaborators demonstrated a taste for very strongly worded allegations that they failed to document.

Underestimation cannot be explained by error and is best explained by strategic misrepresentation, that is, lying. The policy implications are clear: legislators, administrators, investors, media representatives and members of the public who value honest numbers should not trust cost estimates and cost-benefit analyses produced by project promoters and their analysts.

A cynic might describe the above as a flagrant strategy to garner attention, name recognition, publicity for upcoming projects — and to discourage criticism. The 2002 paper attracted much attention and stirred significant commentary, but little formal criticism followed. This does not surprise us. Many academics have a strong vested interest in their pet theory — that costs are systematically underestimated and utilization overestimated for large public works projects — and have little interest in criticism or “alternative” explanations. This, as we demonstrated in a previous post, has led to a situation where a paper with significant analytical flaws (Flyvbjerg et al. 2002) was followed by one with even greater flaws (Flyvbjerg et al. 2005). Nonetheless, a “rerun” of the 2002 media frenzy is all but certain.

Here’s the abstract from the 2002 JAPA paper:

This article presents results from the first statistically significant study of cost escalation in transportation infrastructure projects. Based on a sample of 258 transportation infrastructure projects worth US$90 billion and representing different project types, geographical regions, and historical periods, it is found with overwhelming statistical significance that the cost estimates used to decide whether such projects should be built are highly and systematically misleading. Underestimation cannot be explained by error and is best explained by strategic misrepresentation, that is, lying. The policy implications are clear: legislators, administrators, investors, media representatives, and members of the public who value honest numbers should not trust cost estimates and cost-benefit analyses produced by project promoters and their analysts.

And the one from the 2005 JAPA article:

This article presents results from the first statistically significant study of traffic forecasts in transportation infrastructure projects. The sample used is the largest of its kind, covering 210 projects in 14 nations worth U.S.$59 billion. The study shows with very high statistical significance that forecasters generally do a poor job of estimating the demand for transportation infrastructure projects. For 9 out of 10 rail projects, passenger forecasts are overestimated; the average overestimation is 106%. For half of all road projects, the difference between actual and forecasted traffic is more than ±20%. The result is substantial financial risks, which are typically ignored or downplayed by planners and decision makers to the detriment of social and economic welfare. Our data also show that forecasts have not become more accurate over the 30-year period studied, despite claims to the contrary by forecasters. The causes of inaccuracy in forecasts are different for rail and road projects, with political causes playing a larger role for rail than for road. The cure is transparency, accountability, and new forecasting methods. The challenge is to change the governance structures for forecasting and project development. Our article shows how planners may help achieve this.

Now, JAPA and the APA may indeed have been wary of publicizing Flyvbjerg’s work — but they obviously weren’t so afraid of him that the journal would refuse to put out a second version of the allegedly controversial article three years later. That would seem to blunt at least some of the Oxford researcher’s complaint.

For its part, the APA gave us this statement:

1. The American Planning Association (APA) finds numerous statements and conclusions expressed by the author of this opinion piece to be without merit or any basis in fact. For example,  APA made copies of the article written by Bent Flyvbjerg,  “Underestimating Costs in Public Works Projects Error or Lie?” published in the Journal of the American Planning Association (JAPA), Summer 2002 (Vol. 68, No. 2), available to any journalist or media organization that requested a copy.  To imply, suggest or state anything to the contrary is completely false.
2.  APA recognized that like many JAPA articles, the aforementioned article had news value but after discussing this internally and with Bent Flyvbjerg, and learning of his own extensive efforts to obtain publicity, concluded that this article did not need additional APA promotion and publicity.  The New York Times, The Boston Globe, The Detroit Free Press and The Seattle Post-Intelligencer were among the newspapers in 2002 that published stories about the study.
3.  JAPA is an editorially independent scholarly journal that uses a double blind peer review process. JAPA published a second article by the author on the same topic (JAPA, Vol. 71, No. 2, Spring 2005) titled, “How (In)accurate Are Demand Forecasts In Public Works Projects?”  APA issued a news release on March 29, 2005, about this article, including a quote from David Sawicki, JAPA editor at the time, about the role JAPA plays encouraging “conversations throughout the world that can lead to improvements and enhancements of our current planning practices.” The news release also quotes Mr. Sawicki as calling the paper a “cautionary tale for all projects.”  JAPA’s publication of the 2005 article and a news release are additional examples that contradict conclusions drawn by Mr. Flyvbjerg in his “Viewpoint” published by Cities.
4. Ethics issues involving members of APA and its professional institute, the American Institute of Certified Planners (AICP), are addressed by the Ethics Committee, one of six standing committees of the American Institute of Certified Planners Commission, not solely by the Ethics Officer. Attorneys including outside counsel, are also involved in the ethics process. Suggestions or implications that there are no checks and balances in the way APA addresses matters involving the ethical practices of its members, and the organization itself, are totally inaccurate.Decisions of the Ethics Officer may be appealed to the Ethics Committee, a committee of the AICP Commission. These appeals do happen and the process is designed to provide fairness to the accused individual as well as ensure that no one person can manipulate the ethics process.

We asked Ali Modarres, the editor of Cities, his rationale for publishing the piece. He said “it is the topic”:

This piece belongs to the Viewpoint section of our journal. This section exists to generate debates/conversations about important topics/methodologies. We welcome responses/commentaries related to these Viewpoints. As space allows, we publish a few of these commentaries along with the original Viewpoints. Please come back a few months from now to see commentaries and responses to this Viewpoint. We hope to have a few of them.

Oh, and we forgot one more thing: Careful readers might recognize a name in two of the references Flyvbjerg lists in his editorial. It’s that of a well-known Dutch social psychologist named … Diederik Stapel — author, of, among other things, a study titled “When nothing compares to me: How defensive motivations and similarity shape social comparison effects.”

In fairness, neither of the two Stapel papers is among the 49 articles of his we know to have been retracted and perhaps won’t ever be. But in a paper decrying purported ethical shortcomings of an institution and its leaders, probably best to leave out the work of a confessed fraudster. Just saying.