Troubled article ranking business schools earns expression of concern

An article that ranked University of Missouri-Kansas City number one in an area of business school training is set to receive an expression of concern. The move follows months of questions over the ranking’s legitimacy, following revelations such as a relationship between the authors and both the school and its top ranked researcher in the field.

In 2011, the business world got a bit of a surprise: In the field of innovation management, the study of how entrepreneurs convert good ideas into profit, the number one school – according to an article in the Journal of Product Innovation Management — was UMKC. Not Harvard, not Stanford, not any other institution that normally tops these types of rankings. UMKC’s Henry W. Bloch School of Management was also home to the number one researcher in that field, Michael Song.

The school, of course, was elated, immediately issuing a press release titled “UMKC Ranked No. 1 in the World.”

But after publication, a UMKC professor raised concerns about the paper’s methodology. An investigation by the Kansas City Star uncovered some issues:

At the newspaper’s request, independent experts analyzed the No. 1-in-the-world study, which was published in the Journal of Product Innovation Management. One concern the experts cited: a previously undisclosed relationship between the university and the study’s Chinese authors. In addition, the experts said, it appears the study may have been structured in such a way to ensure that the Bloch School received the top ranking.

(Our own Ivan Oransky weighed in, analyzing the paper for the Star, telling the newspaper: “I just think this paper is fatally flawed.”)

The authors — Pianpian Yang and Lei Tao — based their rankings on the number of articles published in 10 journals they determined to be “top” in the field. The paper — which has never been cited, according to Thomson Scientific’s Web of Knowledge — also credited authors based on where they were at the time, rather than where they did the work that led to these high-profile papers. Yang and Tao listed their affiliation as Xi’an Jiaotong University in the paper; Yang is now based at Chongqing University.

The journal asked “three eminent scholars” in innovation management to independently review the article, and they each “agreed that the method employed was acceptable,” according to the journal’s EoC note.

However, as a result of an audit by PricewaterhouseCoopers published in January of this year, the journal learned that the authors were visiting scholars at UMKC between August 2010 and August 2011. And the top ranked professor – Michael Song – may have contributed substantially to the article. According to the PwC audit:

He “may have written parts related to the strategy portion of the paper” beyond basic editing and grammatical changes.

Song has since resigned from UMKC.

These findings prompted the journal to issue an EoC for “Perspective: Ranking of the World’s Top Innovation Management Scholars and Universities,” published in JPIM, Volume 29 (2), 2012. As a “Perspective,” the paper was not peer reviewed before publication.

We received this statement from UMKC:

We certainly respect the point of view of the Journal of Product Innovation Management, just as we respect the findings of Dr. Robert Hisrich, who in his independent review found that the article, and the methodology used to create it, were consistent with generally acceptable professional practices. Today, UMKC and the Bloch School are focused on moving forward, with new leadership in place, to address the problems identified in the Hisrich report. In fact, the Journal’s statement adds nothing new; the issues already have been identified and the focus now needs to be on the work already underway to improve practices and procedures to ensure that this single chapter of the Bloch School’s long and proud history remains an important lesson from the past, and not an element of the future.

We also obtained a document from last year with more of the school’s response to the controversy. (RW readers will especially enjoy its take on Ivan on page 7.)

Here’s more from the EoC notice, which also criticized UMKC for using the study to boost its standing:

Initial concerns were raised by a professor at the University of Missouri, Kansas City (UMKC) about the methodology of the paper. JPIM enlisted the services of three eminent scholars very familiar with the journal and previously active in the field of innovation management to provide an independent review of the article. The three reviewers independently agreed that the method employed was acceptable. More recently, in his review of an independent audit by PricewaterhouseCoopers LLP, Robert D. Hisrich concluded that “the circumstances surrounding the publication of the JPIM article and the methodology employed were consistent with generally acceptable professional practices”.

However, unbeknownst to the Editor during the submission, review and publication process, the audit also uncovered that the authors of the article were former Visiting Scholars at UMKC. Nor was the Editor aware that Professor Michael Song of UMKC “may have written parts related to the strategy portion of the paper beyond basic editing and grammatical changes”. These recent revelations are the motivation for this expression of concern.

JPIM also expresses concern over the use of the article by UMKC. JPIM does not endorse or agree with the statements and interpretation made by UMKC about their ranking in entrepreneurship or innovation, citing this article as evidence. The authors in the article simply found that scholars currently working at UMKC had the highest total number of articles published on innovation management of all universities with scholars publishing in innovation management, using the method they fully disclosed in the article.

We’ve sent a request for comment to both authors, and will update if they respond.

Like Retraction Watch? Consider supporting our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, and sign up on our homepage for an email every time there’s a new post.