Less than 24 hours after OpenAI posted over 700 preprints describing major math results by an internal AI model, the company has withdrawn three of the manuscripts for errors.
On Oct. 6, OpenAI published 722 preprints in a GitHub repository about purported progress in solving 372 math problems in geometry, computer science, algebra and other subjects. The company called the release an effort “to push the frontier of human knowledge and enable further progress in mathematics.”
But on Oct. 7, OpenAI announced the withdrawal of three manuscripts because of a sign error. The error invalidated an argument in one manuscript and the construction used by two dependent papers, according to the announcement. Each of the withdrawn papers carry notices explaining “the gap,” and that OpenAI has “revised 14 other manuscripts with proof repairs, corrected statements, clearer hypotheses and dependencies, and one correction to an obsolete citation.”
That an error cascaded into three of the OpenAI manuscripts is not surprising, said Alex Townsend, an associate professor of mathematics at Cornell University in Ithaca. He suspects there will be more errors found, he said.
“Given the skepticism of AI in the mathematics community, I think that OpenAI should have announced the manuscripts that were lean verified first,” Townsend told us, referring to Lean, the open-source programming language that checks math proofs. “They could have made a separate announcement for the other ones and ask for help verifying them. This would have been more clear to the community and general public.”
A spokesperson for OpenAI told us the company identified the errors during an audit. As with “other research manuscripts,” the process is iterative, the spokesperson said.
“We welcome scrutiny and feedback from the mathematical community,” the spokesperson told us “Where errors are identified, we will work to correct them promptly and withdraw papers if no fixes can be found. We want to give the community time and space to assess the work, and we’re continuing to improve the clarity and presentation of future papers.”
In a post on X, Dan Roberts, a research lead for the company, announced the withdrawals and wrote OpenAI will “continue to update the repo with new formalizations and with any errata we notice.”
The OpenAI spokesperson told us the company’s collaborators at the Advisory Group on Mathematics and Artificial Intelligence recommended releasing the results without waiting for “full formalization,” and that roughly 50% of the results were released unconfirmed.
“We hope to learn from feedback on this release and work with the AGMAI to understand if they have updated recommendations for release protocol moving forward,” the spokesperson said in an email. “Our intent is to follow their protocol as best we can.”
Andrew Sutherland, a senior research scientist in the department of mathematics at the Massachusetts Institute of Technology, said he appreciated OpenAI acting quickly after finding the errors, calling it “the responsible thing to do.”
However, Sutherland emphasized that many mathematicians are “unhappy with how OpenAI has behaved up to this point,” especially in its handling of the Sept. 8 announcement an AI system had solved the Navier-Stokes problem, which asks whether a series of equations describing how gases and liquids move become unreliable at some point.
Following the release, a group of leading math researchers expressed concern that OpenAI rushed the Navier-Stokes announcement “leaving no time for a proper writeup, the isolation of new methods and ideas, and citing relevant previous work of others.” More than 8,000 researchers have endorsed the sentiments.
“I think the withdrawals and corrections will be viewed positively by the mathematics community, but it will take a lot more than that to earn back the trust they have lost,” Sutherland told us.
The Association for Human Mathematics criticized the most recent release as well. “Mathematicians did not ask for this work to be done,” the group’s statement, released Oct. 7, said. “Releasing over 700 files at once is not a demonstration of scholarship, but a demonstration of power. We urge mathematicians to discontinue their work with OpenAI and to return to a vision of science that centers human understanding.”
Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on X or Bluesky, like us on Facebook, follow us on LinkedIn, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].
Vogon poetry
The ‘pleasure’ is always in the understanding.
Now it will be in understanding AI generated proofs
Sort of right maybe most of the time is not the standard that should be applied to either human generated or AI generated solutions.
A proof is a proof no matter who or what made it. I have seen mathematician gender discrimination and age discrimination. This AI bias is just another form of continuing discrimination.
AI is welcome to write proofs. But AI must expose whose results AI uses to prove. AI should also be able to identify if AI rediscovers human’s results in its proofs, and tell us if that is the case. The same is applied to humans, if they used AI. AI is welcome to check human’s work, but humans should not be forced to do the same for AI’s work. The latter should be optional. This is not discrimination but because AI does not eat or sleep or need to take rest or conscious or need to make a living.
Another perspective is appreciating they released these preprints instead of keeping them and building on private benefits.
Solving all these problems and not communicating would have been acceptable? Releasing a preprint is an invitation to scrutiny and collaboration.
The released preprints cannot be used to make money if kept in private vaults. They need humans to check for free, and their primary intention is to make profit when others are hooked to their AI.
Now more and more mathematical problems are stained by AI, and no mathematicians will work on these in fear of AI plagiarism which is never officially sanctioned.
Their so-called Lean verifications aren’t holding up either.