Artificial intelligence is not yet capable of replacing human mathematicians, according to the results of the “First Proof” challenge – a rigorous test designed to assess the ability of large language models (LLMs) to conduct original mathematical research. Released on Valentine’s Day, the challenge presented ten complex “lemmas” (minor theorems) to AIs, tasks typically assigned to gifted graduate students. The outcome? No LLM solved all ten problems independently.
The Challenge and Its Purpose
The First Proof initiative, spearheaded by eleven leading mathematicians, aimed to push AI beyond regurgitating existing techniques. The problems were designed to demand genuine originality, forcing LLMs to synthesize new solutions rather than simply remixing known ones. This test underscores a critical reality: while AI excels at pattern recognition and data processing, it still struggles with the creativity and abstract thinking that drive mathematical breakthroughs.
Unexpected Engagement from AI Developers
The challenge unexpectedly drew significant attention from AI companies like OpenAI, which deployed substantial resources to tackle the problems. Mohammed Abouzaid of Stanford University, a member of the First Proof team, noted, “We did not expect that the AI companies would take it this seriously and put this much labor into it.” This highlights the growing competition within the AI industry to develop models capable of genuine mathematical reasoning.
Results: Confidence Doesn’t Equal Correctness
The First Proof team revealed that LLMs confidently produced proofs for all ten problems, but only two were verified as correct. One of these proofs had already been documented, and another was partially derived from an archived sketch by a renowned mathematician. Furthermore, many submitted solutions proved to be convincing but ultimately flawed, underscoring the difficulty of distinguishing between genuine insight and AI-generated plausibility.
A Glimpse into AI’s Mathematical “Style”
Interestingly, the correct solutions generated by AIs exhibited a distinctly 19th-century mathematical approach, according to Abouzaid. This suggests that while AI can mimic established methods, it has yet to evolve toward the cutting-edge techniques defining modern mathematics.
The Future of AI in Mathematics
The First Proof experiment is not just about failure. It’s a learning opportunity. The team plans a second round with stricter controls, indicating a commitment to refining the methodology and pushing AI further. Despite current limitations, the rapid progress in LLM capabilities suggests that AI will continue to play an increasing role in mathematical research. Some mathematicians believe AI-assisted tools are already poised to change the field, as noted by Scott Armstrong of Sorbonne University: “These tools are coming to change mathematics, and it’s happening now.”
The First Proof challenge reinforces a crucial point: while AI can accelerate certain aspects of mathematical work, it has not yet achieved the independent, creative reasoning necessary to replace human mathematicians.
