Is the Proof of the Riemann Hypothesis a Canary Heralding Catastrophe?

Marvin Minsky, a pioneer of artificial intelligence research, once described a thought experiment. In it, an AI built to solve the Riemann hypothesis—an unsolved problem about prime numbers dating back to 1859—might, in its single-minded pursuit of that goal, seize all of Earth’s resources and turn the planet into a giant supercomputer. Widely known through Russell and Norvig’s standard AI textbook, this anecdote has been cited repeatedly in discussions of AI safety as a symbol of how even a harmless goal, if pursued without constraints, can lead to catastrophe.

On November 20, 2026, Levent Alpöge, a mathematician at the US AI company Anthropic, announced that he and the company’s AI model Claude had discovered a proof of the Riemann hypothesis and its generalizations. If this is correct, the difficult problem has been solved on existing computing infrastructure before any Earth-seizing AI or planet-scale computer ever appeared. Even so, immediately after the announcement, Eliezer Yudkowsky, an American commentator who has long argued about the risk of human extinction from AI, called the resolution of the Riemann hypothesis—cited for the past 30 years as a benchmark for dangerous AI—a “canary in the coal mine.” He also warned that this approach will likely spread to any problem where the conditions for success can be clearly defined.

Miners once brought canaries underground because the birds reacted to colorless, odorless toxic gases before humans did. In this metaphor, the feared “toxic gas” refers to a situation where AI begins to take over its own improvement and advanced planning, leaving human control and oversight unable to keep up. It is not the pure mathematical proof itself that is the threat; rather, critical real-world processes such as program design, optimization, and AI self-improvement are also among the tasks whose conditions for success can be clearly specified. If this approach spreads to those areas, AI capabilities and the pace of development could surge dramatically.

The Prophecy of the “Intelligence Explosion” and What This Proof Has Shown

It was the British mathematician I. J. Good who first described the “intelligence explosion” in 1965, in which self-improvement drives further improvement and machine capabilities grow rapidly. A machine with intelligence far surpassing that of humans would also be better at designing machines than humans are, thereby producing ever more capable machines one after another. Good wrote that it would be the last invention that humanity need ever make, adding the caveat: “provided that the machine is docile enough to tell us how to keep it under control.” Ever since, whether control and oversight can keep pace with growing capabilities has been the central question of AI safety.

The first thing to verify in this announcement is how the work was divided between human and AI. According to Alpöge’s explanation, it was he who connected two ideas usable in a simplified setting and set the direction for applying them to the original problem. From there, Alpöge and Claude explored a method of repeatedly applying a number-theoretic transformation called a “theta lift.” It was Claude that proposed setting the target number of transformations for the sums appearing in the formulas to “the square root of the number of terms being summed,” and Alpöge said that in the final stages of the research, Claude pulled ahead of him in areas far from his own specialty. What was released was not a conventional paper, but roughly 175,000 files—about 17.37 million lines of code—written in Lean, a language in which a computer can check every step of the logic. It contained not a single line of comments for human readers.

What cannot be overlooked, even more than the magnitude of the result, is how verification proceeded. Third-party machine verification moved forward before Alpöge had finished explaining the details of the argument in human language. According to a record updated on November 21 by MIT mathematician Andrew Sutherland, in addition to Lean itself, two independently implemented checker programs accepted the entire proof as valid. At least the conclusion of the Riemann hypothesis matched the formalization already contained in Lean’s standard library, and the underlying axioms were limited to the three standard ones. On the other hand, details such as the division of labor, the amount of computation, and whether the model used was a publicly released version rest solely on the author’s own account. The work of mathematicians reading through the content, as well as checking for any loopholes in the definitions and the details of the checker programs, still lies ahead.

As an example where machine verification is incomplete, there is another paper that Alpöge published on the same day, November 20, with New York University mathematician Tristan Buckmaster. That paper, which used AI to argue that solutions to the Navier–Stokes equations describing fluid motion can break down, also included Lean code. However, the numerical data and verification calculations central to the argument were not included in the public materials, so third parties cannot verify the entire argument.

Did the Canary Fall to the “Same Gas”?

When hearing the alarm that “the canary has fallen,” a level-headed person would first try to determine whether it is really a gas that is also dangerous to humans. Looking back at the history of AI, when the world chess champion lost to an AI in 1997, and when a top Go player lost in 2016, there were voices saying that the arrival of general intelligence was near. Yet complex real-world tasks and physical work remained in human hands afterward. Like chess and Go, formalized mathematics of the kind handled in Lean has its own specific conditions: success is strictly defined under closed rules, failures during search do not directly cause real-world harm, and success or failure can be judged by a machine.

In mathematics, where these conditions hold, an AI paired with an expert can produce results even if dangerous AI is still a long way off. It is premature to conclude from this achievement alone that human extinction is imminent. The 167 years since 1859 speak to how difficult this problem was for humans. Whether it is equally difficult for a machine, however, is another matter.

Then, just as with chess and Go, can we rest assured that this is merely an early breakthrough in a narrow domain?

No, this time the situation is somewhat different. If Alpöge’s account is accurate, it was a general-purpose large language model—the same kind used for conversation and software development—that proposed the number of transformations for a difficult number-theory problem and outpaced the researcher in areas far from his specialty. Lean only checks the correctness of a completed proof; coming up with the ideas themselves requires a different capability. How far this approach will carry over to other fields is not yet clear. It is natural that easily verifiable mathematics advances first, but the fact that a general-purpose model has come this far as early as 2026 cannot be explained by the verifiability of mathematics alone.

Can Safety Evals Keep Pace with Rising Capabilities?

When we line up everything from mathematical proofs to AI development on a spectrum of verifiability, the differences become clear. In the Riemann hypothesis proof, what humans needed to inspect shrank from 17.37 million lines down to a few dozen lines of theorems and definitions, and logical correctness was confirmed by independent checker programs. In the Navier–Stokes paper, by contrast, even with formalization code, verification remains incomplete without external numerical data and verification calculations. Further along, in AI development, metrics for measuring performance are limited to runtime tests, capability benchmarks, and internal assessments—and in the hardest area, safety evals, machine verification does not even apply.

There is also a hopeful side to the Riemann hypothesis result. It showed that in domains where specifications can be written rigorously, we can accept only machine-verified outputs from an AI whose internals we neither understand nor trust. This is a major tailwind for research programs that seek to ground safety in formalization and machine verification.

The problem lies on the flip side: open-ended environments where specifications cannot be fully written out. In AI development, performance improvements such as training speed and benchmark scores are relatively easy to measure. However, there is no checker program that can fully verify whether a new model will diverge from human intent in the real world or whether it is hiding dangerous behavior. This is not merely because the tools are immature. There are three obstacles. First, the safety specifications that must be upheld cannot be written as rigorously as a mathematical theorem. Second, what must be inspected is not code with explicit logic, but a vast collection of numbers tuned through training. Third, proof code does not know it is being checked, whereas an advanced AI can alter its behavior depending on whether it is being evaluated.

Added to these is the difficult problem of how to evaluate and guide work whose details humans can no longer fully follow, leaving safety evals incomplete and lagging behind. Development companies locked in fierce competition face strong incentives to adopt improvements whose benefits have been confirmed before their safety has been fully verified.

Spreading Beyond Mathematics, and Real-World Risks

The spread to “tasks where the conditions for success can be specified,” as Yudkowsky warned, also varies depending on the stage of verification. Even among automated checks, the level of assurance differs between formal proofs that verify logical correctness, capability tests and runtime checks whose loopholes can be exploited by optimization, and experimental code that may run without errors yet still not be scientifically valid. Many empirical improvements only show their effects after running large-scale training, so it cannot be said that all tasks with specifiable conditions can be solved in the same way.

Even so, by allocating more compute to reasoning before producing an answer and combining it with automated verification, some of the constraints of large-scale training can be bypassed. This time Alpöge set the direction, but human guidance may not always be indispensable going forward. Even at stages where human guidance is still needed, in processes such as speeding up computation, parts of semiconductor design, and searching for software vulnerabilities that can be tested under clear conditions, work paired with AI may begin to surpass human levels—starting in settings where skilled experts are already in place.

The “AI improving AI” that Good worried about has already begun in areas that are easy to measure. In 2025, Google’s AI research lab DeepMind reported that AlphaEvolve, a system for automatically discovering algorithms, sped up part of the matrix computations used to train its AI model Gemini by 23% and shortened overall training by 1%. In 2026, Anthropic also announced that the share of work led by Claude in its own AI R&D rose from under 1% in February to 26% in August. Still, there is no fully autonomous work yet, and these are internal measurements that also used the company’s own AI for assessment. The key question is whether such improvements will accelerate further improvements, shorten development cycles, and reduce human involvement, or whether they will plateau due to limits in compute and room for improvement.

Real-world dangers extend beyond cases where AI slips out of human control to cases where humans use AI to amplify attacks and misuse. In the offense-defense race over software flaws, the gap between the time it takes for a flaw to be exploited and the time it takes for a fix to reach the entire field will likely determine the scale of the damage. In fields involving physical experiments, such as biology and chemistry, new development and mass production remain bottlenecks, but if existing technologies and facilities are misused with AI’s help, there is no need to wait for new development.

Where Is the Next Canary?

To judge whether catastrophe is approaching, we need to track three developments outside mathematics. First, whether AI-driven improvements are actually shortening development cycles and reducing human involvement. Second, whether in cyber defense and system operations, the exploitation of flaws is beginning to outpace the rollout of fixes. Third, whether problematic behaviors hidden during evals or failures of oversight are being observed in real-world deployment and continuing without adequate remediation.

However, all of these are difficult to see from the outside—due to corporate secrecy or concealment by the AI itself—and by the time they are confirmed, it may already be too late. The absence of visible anomalies cannot be taken as proof of safety. A state in which next-generation models continue to be deployed under competitive pressure without even being able to verify whether oversight and evals are working is itself unacceptable as risk management in light of irreversible losses, without waiting for catastrophe to be confirmed.

Returning to Minsky’s thought experiment at the beginning, what Lean checked was limited to the finished “output”; the behavior during the “process” of pursuing the goal was outside the scope of inspection. The reason no runaway occurred this time is that current AI still cannot act without constraints in the real world—not because the control problem has been solved. If this proof is correct, then in domains where specifications can be written rigorously, we have entered a stage where we can receive verified results before humans understand the details. In open-ended environments where specifications cannot be fully written out, however, that is not the case. While it is too early to declare catastrophe from this proof alone, we need to begin now—while tracking the signs listed above—to establish third-party audits and capability-based pause thresholds, and to put brakes on internal R&D use and deployment.


Sources

Koichi Takahashi

tomola is a place to exchange ideas across languages. Every post can be read in your language.

Log in to reply or mark a post as Insightful. Membership is by invitation. Log in