A group of authors at a Pittsburgh company have proposed a new way to write, review, and read scientific papers that they claim will “radically alter the creation and use of credible knowledge for the benefit of society.”
From the abstract of a paper appearing in the new Mary Liebert journal Disruptive Science and Technology, which, according to a press release, will “publish out-of-the-box concepts that will improve the way we live”:
We developed a new literature format based on an interactive network to address the needs of all parties, from author to user. We began by structuring the writing of text and data for a discipline’s needs. Five report types were created, with menus for specific terms and data to allow online, simultaneous, multiauthor writing and editing. A new measurable peer-review process was created. Users can ask questions of reports, and data from multiple reports can be combined. A topic search is associated with automated research tools. We built a prototype that was built and refined based on continuous feedback from surveys, expert panels, presentations, and other feedback mechanisms. In continuous development, the free network World Science (www.world-sci.com) was launched for global beta testing in April 2011, and is now used worldwide. Reports have been written, reviewed, and published within the network or in other journals.
One of the aspects of World Science that caught our eye is that it would allow users to ask questions of articles they were reading. From the site’s FAQ:
9. How do I ask direct questions of articles?
For any article published in World Science, you will see a list of interactive features below the title. Click on the drop-down menu for a list of available questions and answers will be provided from the text of the article.
We were particularly interested in how World Science would change peer review. Also from the FAQ:
10. Why is the peer-review process of World Science unique?
The process is unique for several reasons. First, reviewers provide their comments inside the article itself, below each of the questions within each section. Authors can then revise their work directly. Second, authors “grade” each of the review comments as to whether the comment was helpful or not, the revision made or not made. The editor can summarize these for each reviewer.
The paper’s corresponding author is Douglas Kondziolka, a professor of neurosurgery at the University of Pittsburgh. Kondziolka and his co-authors are all founders or consultants to SciencEngines, Inc., which created World Science. They make bold claims about their proposal:
We believe that this format will radically alter the creation and use of credible knowledge for the benefit of society. The technology is disruptive to the current publication model and creates new learning, research, and professional opportunities.
So we asked a few leading journal editors, who’ve justifiably made a name for themselves cleaning up the scientific literature, for their takes on what this would mean for journals. Steven Shafer, editor-in-chief of Anesthesia & Analgesia, told us:
There are numerous novel approaches to publication that challenge traditional publication. All of them use Internet technology to blend authors, readers, and reviewers. The extraordinary success of Wikipedia, and the ensuing demise of the traditional printed encyclopedia, is an example of how a disruptive technology can quickly supplant an established paradigm.
The proposal by Kondzoilka is an experiment. It will succeed if it advances the creation and dissemination of new knowledge. It will fail if it offers nothing beyond what is currently captured in the Wikipedia model of publishing. The hundreds of millions of authors and readers who turn to the Internet daily will determine the outcome of the experiment. The experiment is worthwhile, and I commend the authors for their innovation.
Ferric Fang, editor-in-chief of Infection and Immunity, also noted the similarity to Wikipedia, but was a bit more dubious. Fang told Retraction Watch he’s seen “an increasing number of proposals for alternatives to conventional peer-reviewed scientific journals (e.g., community-based evaluation, blinded review).” He says this one
sounds rather like a wiki, which can be very useful as a repository of conventional knowledge, but I am skeptical that it would function well for actual science. The proposal seems to ignore many of the current functions of the scientific literature, such as the filtering and prioritization of ideas, establishment of priority, storage of primary research data, and measurement of productivity. One need only to consider that there are in excess of 50,000,000 research papers to realize the importance of the filter function. Curation and moderation of a scientific knowledge network would be a nightmare. How would one prevent contributors who disagree with each other from repeatedly editing the other’s entries?
He continued:
I recall Churchill’s famous comment about democracy– peer review might be the worst way to manage the scientific literature, except for all the alternatives. Although, as you know, I would like to reform science, I guess I am not a revolutionary.
seems pretty similar to the existing format on webmedcentral.
It seems to me that journals are a model that were excellently suited for the days of print based journals but have only grudgingly moved on with technology. To some extent they have embraced technology, but they have not reduced their costs consummerately or expanded access. Partly because it is a money-making business that is monopolised by a few players and partly there are a lot of people who obtain jobs or influence from it and have a vested interested in maintaining the present model.
Publications, in modern science, really represent a sort of league table by which the players can rank their acheivements against each other in competing for resources or positions. If the new system lends itself to obtaining these ladder points, then scientists will use it, but from a love of sharing knowledge alone – well, I am sceptical.
I wonder how well the peer review system works in practice. How often have publications have been fundamentally altered by the additional experiments demanded or even withdrawn altogether, or do the results generally just agree with the original thrust of the paper? Clarification experiments will often only work if they are performed in the hands of people genuinely interested in falsifying their own theories. Likewise, how many excellent suggestions for confirmatory experiments in the new wiki format would actually be acted upon by the original lab.
Meh. Sounds like any of the hundreds of other interactive efforts out there (semantic Biochemical Journal, Elsevier’s “paper of the future” etc.) If there’s only one rule in shaping the future of a given field, it’s something like”get there first, and if not then get there shortly after and do it much better”. Examples would include Wikipedia, Facebook, Google, Apple. This effort is not first, and it certainly doesn’t come across as better.
As with all these types of technology, it requires a certain level of buy-in from the end users, and those users (scientists) could not give a hoot whether the former Skype CEO is on the front page touting how great it is. What matters more is whether my colleagues are using it and whether my article will be read? Is it on PubMed? Does it cost anything? What are the archiving guarantees if ti goes belly up? The fact that a beta-invite is required just to even look, is not encouraging (anything requiring me to fill out an application form giving up personal information and email address is a barrier). Add in the fact that a quick search for my own name brings up 25 papers (from >85 total), and only about half of those listed are mine (it applies a Boolean AND to first name, last name, so papers show up with one author having the same last name as me, and another having the same first name as me). It only goes as far back as 2002. It’s been in beta since April 2011 and is now “used all over the world”. Really? How come I never heard of it until now (and Google Trends hasn’t either)? Do I need to go on?
Oh, and just for good measure, the papers published on there so far by Kondziolka bear STRIKING resemblances to many of his published works (>450 of them) listed on PubMed. Just one example… http://www.cns.org/publications/clinical/54/pdf/cnb00107000241.pdf
vs. http://www.world-sci.com/read.aspx?id=29
Hint… if you’re gonna launch a new journal, don’t use it to re-hash your old papers!
I think that the main claims of a paper need to be tested experimentally (real experiments in the real world) by people who are independent (not in the same management structure, and not having published with the authors of the paper) from the authors of that paper. Ideally the people who test the main claims of the paper should do their work blind, i.e. not be told what to expect. Most discussion about a paper will be heavily influenced by the belief system in that field. Biology is blessed with people whose temples consist of cartoons with “arrows of authority” and gangs of supporters. Blind experiments would have a chance of cutting through the politics.
Your suggestion will work in an ideal world, but real world scenario is different. My publication was rejected by someone working in a related field and in a month the same person published his / her work that is directly related to my (rejected) material. In such competitive world, no one will objectively reproduce other’s work just to publish them.
Further, think of the special equipments / training needed in science and technology to check the experiments. Reproducing results related to health care that have been gathered over months and years is just not possible.
Great stuff! I’ve just recently been writing about ways publishing might change, and there’s a great connection here. Cf. New approaches to quality control in publishing, at http://curt-rice.com/2011/12/06/new-approaches-to-quality-control-in-publishing/
It starts:
There are four key components to publishing, and they’re all about to change.
Ten years from now, publishing will be done in ways that we are only beginning to envisage. Politics and profit will of course compel these changes. But the specific innovations coming our way will be driven by a generation of tweeters, bloggers, status updaters and Wikipedia editors.
In reply to curtrice May 14, 2012 at 5:26 pm
You blithely wrote “Politics and profit will of course compel these changes.”.
You might just as well write “Politics and profit will of course repel these changes.”
I don’t see politics and profit changing anytime soon.
Me, neither. I didn’t say profit and politics will change. I said they will compel change. As they always do.
I just *love* these sites, which state that they are quote: employing “state-of-the-art techniques and technologies to protect the identities of its users.” And then send out a confirmation email containing your chosen password in clear text. Obviously these people know f***-all about data security.