Two years of stonewalling: What happened when a scientist filed a public records request for NASA code

Retraction Watch readers may know Nathan Myhrvold, who holds a PhD in physics, as the former chief technology officer at Microsoft, or as the author of Modernist Cuisine. They may also recall that he questioned a pair of papers in Nature about dinosaurs. In that vein, he has also been raising concerns about papers describing the sizes of asteroids. (Not everyone shares those concerns; the authors of the original papers don’t, and astronomer Phil Plait said Myrhvold was wrong in 2016.) Last month, Myhrvold published a peer-reviewed paper as part of his critique. The final version of that paper went live today, as did a story about the science in The New York Times and a detailed explanation by Myrhvold in Medium. As the discussion over the results continues, here he shares his experience trying to obtain details about the methodology the authors used.

Two years ago, I uploaded a preprint to arXiv.org describing what I considered serious problems, including apparently irreproducible results, that I had uncovered when analyzing a set of research articles published by the NASA Jet Propulsion Laboratory (JPL) NEOWISE project. NEOWISE is the largest scientific analysis of asteroids ever conducted; the researchers on the project have so far published estimated sizes of more than 164,000 objects in the solar system, estimates they have claimed were all derived by applying a standard approach to raw observations from the WISE space telescope.

My findings generated quite a stir in the media, including stories in The New York Times, Science, and Scientific American, among other outlets. My hope and expectation was that shining light on these troubling issues would spur the JPL researchers to retract or correct their papers. At the very least, I thought, they would release the various unpublished techniques that they had used in a series of highly cited papers, stretching from 2011 to 2015, thus lifting the veil of secrecy that had prevented me and other astronomers from replicating their results.

Instead, the researchers dug in their heels. They ridiculed me in the press, telling Science that as a former chief technology officer of Microsoft I was “responsible for a lot of bad software” and asserting to Scientific American that “his math is just wrong.” Yet they refused to defend their suspect choices. And they dismissed my findings because (just like thousands of other astronomy preprints on arXiv.org) my manuscript “was posted before undergoing the essential scientific peer-review process to catch and remove significant errors.”

As it turns out, my math was not wrong, and my findings were up to the challenge of intensive peer-review. Part one appeared in the prestigious planetary science journal Icarus in December, and the larger part two was published last week. Key sections also appeared, after peer review, at planetary science conferences in 2016  and 2017.

NASA’s response got me wondering why the NEOWISE researchers were being so studiously recalcitrant. Typically, when you point out to scientists that they have goofed, they do one of three things: they say “oops!,” they ask you for proof of their error, or they bend over backwards to justify why they did things in an unconventional way. Instead, NASA issued a vague press release berating me with claims I had made a calculation error. (It was actually just a typo – and a red herring. Even if my math had been completely off, that wouldn’t explain or excuse evidence that some of the NEOWISE papers had passed off asteroid diameter measurements made by other researchers as asteroid model results that they had calculated themselves.)

Moreover, the principal investigators of the JPL group, Amy Mainzer and Joseph Masiero, continued refusing to respond to any of my scientific questions, hiding behind a wall of silence that they had thrown up months earlier when I started inquiring about how they had obtained their results and asking for crucial details missing from the published descriptions of their methods. At an asteroid science conference, I tried to talk politely with Masiero, but he flat out refused, saying he wasn’t “allowed” to speak with me.

Turning to FOIA

With the normal channels of scientific communication closed, I turned to the Freedom of Information Act (FOIA), the law frequently used by journalists to pry correspondence, government reports, and internal working documents from federal agencies. By my thinking, basic scientific research at a federal research center like JPL, including all of the related internal reports, presentations, software specifications and correspondence of JPL researchers with outside scientists, should be available for the asking.

So I asked. In June 2016, I filed a FOIA request with NASA and JPL for materials related to the NEOWISE project. Both NASA and JPL immediately bounced my requests. They were “unable to process” them, they said, because “it is unclear what specific records you are requesting.”

Really? One of the requested categories on my list was “Documents about WISE/NEOWISE data analysis, model fitting and details thereof, including any documents on least-squares algorithms, for example the Levenberg-Marquardt algorithm or variations thereof.” That was not specific enough?

Frustrated, I hired some attorneys to revise the request into a form that met all legal requirements. My lawyers submitted very lawyerly clarification letters a few weeks later.

Incredibly, NASA persisted in its claim that it could not process the requests, going so far as to “close” the cases. Among other absurdities, NASA claimed  that it could only search paper files and not email. They can send men to the moon but…never mind.

An appeal

When government agencies fail to honor their obligations under FOIA to fulfill requests like the ones we filed, there is an administrative appeal process you must go through before getting a court involved. So the attorneys filed a formal appeal. A decision is supposed to arrive in 30 days or less, yet months passed with no reply. Finally, in November 2016, a higher official at NASA confirmed that my central requests were proper and that the agency had no basis for its position. The FOIA staff was ordered to search for and produce all responsive records.

Okay, here we go, I thought. Finally, I’d get a peek behind the curtain and could see exactly how the NEOWISE team had arrived at all of those tens of thousands of asteroid diameters that they had released, but that neither I nor any other researcher had been able to reproduce.

But Christmas never came. Over the next few months, NASA trickled out dribs and drabs of documents. Many of them were simply reprints of journal articles and other things already in the public domain. Other documents were heavily redacted , to a ludicrous extent considering that this research has no connection to national security or law enforcement. It was a thousand-plus blank, blackened, and otherwise useless pages almost perfectly selected to avoid touching on the scientific information I had asked for.

Another appeal

And this time, in an ironic twist worthy of Orwell, NASA said it would search only email, and not any other data systems (such as file servers), nor any paper documents. So, in December 2017, my lawyers were forced to file yet another administrative appeal, to force the agency to comply with the judgement it had lost from the previous appeal.

Once again, NASA’s front office came down largely on my side: it decided the FOIA staff were not fulfilling the agency’s statutory obligations, and it ordered the staff to redo the most important parts of the search the right way, and “expedited.” The results of this expedited, more thorough search: several more months of delay and a scant 50 additional pages of documents.

These pages did not include my main goal – the details necessary to replicate the NEOWISE results.  Combing through these combined results for anything useful, we did find an email thread involving Thomas Statler, a high-level program scientist at NASA who has expertise in the thermal modeling of asteroids to work out the objects’ likely size. I had met Statler at an astronomy workshop, and as I was writing my article in 2016, I sent him drafts, on which he had provided many helpful comments. (I had also sent these drafts, and requests for clarifications, to Mainzer, Masiero, and other NEOWISE investigators, but received zero responses.)

In the FOIA response was an email from Statler to Amy Mainzer, sent a few weeks before my preprint came out. “I read Nathan’s previous draft and gave him some pointed suggestions without trying to referee the paper,” Statler wrote. “He is doing some things correctly and other things incorrectly. In the end I see his basic results as largely confirming earlier results in broad brush, but with some modifications. I think that his calculations on the error distributions deserve to be examined carefully.” Apparently intrigued by the anomalies I was finding, Statler asked to see the NEOWISE code.

When Statler asked Lindley Johnson, NASA’s Planetary Defense Officer, whether he should speak to a New York Times reporter about my paper, Johnson responded: “You are ‘not available’ today.” Statler was not quoted by any of the many media who covered the story.

Later in 2016, Carrie Nugent, one of the NEOWISE investigators, emailed a colleague, alarmed that Statler wanted to see the actual code used to calculate asteroid diameters. “Tom has made some unusual requests,” she wrote, for “unpublished diameters [and] the NEOWISE proprietary code. I believe he may be working on a manuscript relating to our diameter derivation. Although I am happy to engage in scientific debate and correct any bugs, I doubt that this paper will be in the spirit of collegial discussion.” Her colleague forwarded the exchange to Johnson, noting “we are having some issues with Statler.”

Moreover, while NASA had been giving me the runaround for 18 months, I had done further digging into the asteroid data that the NEOWISE group had uploaded into JPL’s Planetary Data System (PDS) in June 2016, shortly after my preprint flagged the multiple problems with their scientific papers. My comparison found that they had scrubbed many of their previous results and replaced them with new values, though they were still attributing them, misleadingly, to their old papers. They hadn’t offered any justification (or even disclosure) of this practice, and the shifting data had potentially large implications for the work other astronomers are doing on asteroids.

A new FOIA request

So I filed a new set of FOIA requests in January 2018 to try to shed light on the Statler episode and the reason for culling and altering the data in the PDS. True to form, NASA tossed me a few sparse pages of mostly redacted emails and a handful of already published scientific papers. And this time they offered a novel rationale for concealing relevant information: some of the data I was seeking, they claimed, are “contractor records” that are exempt from FOIA. So I have now filed yet another administrative appeal—the third.

This cannot possibly be what Congress had in mind when it enacted FOIA 52 years ago to bring transparency and sunshine to government operations. Scientific research paid for by taxpayers and performed by government agencies belongs to all of us. Two years of stonewalling, hundreds of thousands of dollars spent on FOIA fees and outside counsel, three administrative appeals: these are the outrageous tolls exacted by a federal agency that impede my quest for scientific clarity.

Like Retraction Watch? You can make a tax-deductible contribution to support our growth, follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up for an email every time there’s a new post (look for the “follow” button at the lower right part of your screen), or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].