Can Transformers Reach AGI? (AGI Theory, AGI and Society, and Japan’s Choice: Part 1)
Introduction
In recent years, artificial intelligence (AI) has been rapidly expanding its “generality”—adapting to diverse situations well beyond specific tasks. Positioned at the ultimate extension of this trajectory is Artificial General Intelligence (AGI).
AGI refers to an artificial system that possesses flexible intellectual capabilities equal to or greater than those of humans, capable of self-directed learning, reasoning, and acting creatively even in unknown environments. However, no unified consensus exists among experts regarding the definition of AGI. For this reason, in theoretical research, instead of directly defining AGI, researchers often employ the concept of Universal AI, which is formulated as an “ideal intelligence capable of acting optimally in any environment.”
Universal AI is a general intelligence model grounded in information theory, with its representative theoretical formulation being the reinforcement learning agent known as AIXI, proposed by Marcus Hutter. AIXI predicts the future based on observations and rewards, choosing actions that maximize expected value. While it provides the theoretical picture of a “completely rational intelligence,” it internally relies on a mathematical prediction principle called Solomonoff Induction, which requires infinite computation and therefore cannot be executed on real-world computers. In other words, AGI can be viewed as an attempt to realize an approximation of this ideal Universal AI using finite resources.
Solomonoff induction is a principle of inference based on algorithmic information theory proposed by Ray Solomonoff in the 1960s, which assigns higher probabilities to shorter, simpler hypotheses (programs) among all those that explain observations. Put simply, it seeks to achieve predictions with the highest generalization capability by “adopting the shortest explanation.” This can be considered a mathematical formalization of “Occam’s razor,” standing as a foundational concept underpinning both the rationality of scientific theories and the learning principles of AI.
Against this theoretical backdrop, the question confronting current AI research can be framed as follows:
How close is the Transformer, the core technology of modern AI, getting to Universal AI?
The current rapid progress in AI capabilities is heavily supported by the advent of deep learning models known as Transformers. The innovation of Transformers lies in having demonstrated “scaling laws,” where performance improves seemingly without bound as model size, data volume, and compute are expanded simultaneously. This empirical law has justified a development race predicated on massive compute and vast amounts of data, ultimately underpinning the evolution of leading AI systems such as ChatGPT, Claude, and Gemini.
Does theoretically defined Universal AI—that is, AGI—truly lie at the extreme end of this scaling? Or must we await an entirely new, foundational paradigm shift?
In response to this fundamental question, multiple theoretical studies published between 2024 and 2025 present a new direction. The common claim among them is that Transformers may asymptotically approach Solomonoff induction (SI) via algorithmic information theory—meaning they could be continuous with theoretical Universal AI. In other words, a perspective is emerging that modern AI is not merely an empirical statistical model, but may be asymptotically approaching the theoretically defined “optimal intelligence.”
Across three installments, this series will examine the possibility that the modern AI development race is continuous with AGI and what that implies, structured around three perspectives:
- Part 1 (this article): Introduces recent learning theory research suggesting Transformer-based models can asymptotically approach Solomonoff induction, showing the possibility that AI and AGI are more closely continuous than previously thought.
- Part 2: Building on the possibility of continuity between AI and AGI, examines the implications of the scaling race led by US and Chinese tech giants, as well as strategic considerations for a desirable future.
- Part 3: Explores the research and policy direction Japan should take, offering several concrete proposals.
Transformer Learning Asymptotically Approaches Solomonoff Induction
The following three papers, published independently between 2024 and 2025, develop the claim that Transformers / large language models (LLMs) approach Solomonoff induction through training, drawing on formal theory and empirical evidence:
- Paper: Learning Universal Predictors, ICML 2024 (arXiv:2401.14953)
Authors: Grau-Moya, Genewein, Hutter, Orseau, Delétang, Catt, Ruoss, Wenliang, Mattern, Aitchison, Veness (Google DeepMind)
Key claim: Published by Google DeepMind with authors including Hutter, an authority on Universal AI theory. Demonstrates both theoretically and experimentally that Solomonoff induction can be amortized into neural networks (Transformers and LSTMs) via meta-learning using “universal data” generated from universal Turing machines (UTMs). Formally shows that in the limit, the meta-training procedure converges to SI, and experimentally confirms that the acquisition of universal prediction strategies advances as model scale increases. It also argues that the in-context generalization ability of LLMs itself can be regarded as a coarse approximation of SI. - Paper: Transformers As Approximations of Solomonoff Induction, ICONIP 2024
Authors: Young & Witbrock
Key claim: Transformers can be compared to the “limit-optimal predictor” of SI, which mixes all computable probability distributions. Posits a hypothesis that frames the predictive performance and scaling behavior of Transformers in correspondence with SI, outlining the arguments for and against it. - Paper: Large Language Models as Computable Approximations to Solomonoff Induction, arXiv:2505.15784 (2025)
Authors: Wan & Mei
Key claim: Proves formal theorems showing that the LLM training process approximates the SI prior, and next-token prediction approximates SI induction. Experimentally verifies a “few-shot learning strategy prioritizing low-confidence examples” derived from that theory, confirming its effectiveness in small models.
(Note: Although there are other studies currently under submission, they are not covered in this article.)
These theoretical studies share common assumptions and caveats. Transformers asymptotically approach Solomonoff induction theoretically only in the limit where “infinite data and compute” are provided; in reality, constraints on time, memory, and optimization methods exist. Furthermore, when model scale is small, complex regularities cannot be adequately captured, and real-world data biases and noise also restrict performance. Nonetheless, few-shot and in-context learning behaviors have been confirmed to align with theory: Grau-Moya et al. (2024) experimentally verified that Transformers/LSTMs trained on UTM data acquire universal prediction strategies as model scale grows, and experiments by Wan & Mei (2025) demonstrated that sample selection strategies derived from theory contribute to actual performance improvements.
Note: We at the AI Alignment Network are also pursuing related research. For instance, in a 2025 paper, we showed that Self-AIXI, which computably approximates AIXI using self-prediction, can asymptotically approach Universal AI through learning. For details, see the following article:
Control of AGI and Curiosity: Publication of the Paper “Universal AI maximizes Variational Empowerment”
Thinking Transformers Will “Not” Reach AGI Is Dangerous
What, then, can be drawn from these theoretical studies?
Previously, there was no clear answer to how far scaling laws would continue or where they would lead. However, recent theoretical research is pointing in a distinct direction: “what lies beyond is Solomonoff induction.” If Transformers asymptotically approach Solomonoff induction through learning, what lies ahead is Universal AI—a concept that theoretically surpasses AGI. This implies that somewhere along the extension of scaling laws, AGI inevitably exists.
Of course, infinite data and compute do not exist in reality, and it remains uncertain whether AGI is contained within reachable bounds. Moreover, numerous constraints remain, such as uncomputability, optimization, finite resources, and the absence of embodiment. In particular, some researchers maintain that “flexible action is impossible without embodiment.”
However, there is one crucial implication. Whereas it was once thought that unknown theoretical and technical barriers stood on the path to AGI, in reality no such barriers may exist—the hurdle might simply be quantitative expansion. The fact that this hypothesis has been proposed independently by multiple theoretical researchers carries significant weight.
Therefore, while whether current AI will reach AGI remains undetermined, we should recognize that asserting it will not reach it has become more difficult than ever. In conclusion, given the immense societal impact of reaching AGI, conceiving the future of society under the assumption that “AGI will not be achieved” is dangerous.
Conclusion
This article is the first installment of a three-part series. In Part 1, we introduced how, based on recent theoretical research, the possibility of Transformers (and LLMs) reaching AGI is shifting from a “distant hypothesis” to a “realistic scenario on the extension of existing technology.”
In the next installment, Part 2, we will discuss the implications of the scaling race led by US and Chinese tech giants in light of the potential continuity between AI and AGI, as well as what needs to be considered to steer toward a desirable future. In the final Part 3, we will examine the research and policy path Japan should take and present several concrete proposals.
(First edition published on October 7, 2025. Updated for technical accuracy on April 16, 2026.)
This article is reprinted from the ALIGN Blog (October 7, 2025). Original: https://www.aialign.net/blog/transformer-agi-part1
Koichi Takahashi