AGI Control and Curiosity: Paper “Universal AI maximizes Variational Empowerment” Released
Yusuke Hayashi (ALIGN) and Koichi Takahashi (ALIGN, RIKEN, Keio University) have released a new paper on the controllability and safety of AGI (arXiv:2502.15820). This blog post explains the contents of that paper.
■ Why is it difficult to discuss AGI? — Introduction
“AGI (Artificial General Intelligence)” refers to the concept of general-purpose AI that possesses intelligence equal to or greater than human intelligence and can perform a wide variety of tasks, contrasting with conventional narrow AI specialized for specific tasks. Given the rapid progress in recent years, some hold the view that AI worthy of being called AGI could be realized within the next several years.
So, on what basis can we say that AGI has been achieved? In fact, there is no settled definition upon which all experts agree. This paper adopts the mathematical formulation of Legg–Hutter intelligence. This captures intelligence as “the ability to achieve goals in a wide range of environments” and quantifies it as the capacity to maximize reward from the environment. An “ideal” AI with maximal Legg–Hutter intelligence is called a “Universal AI.” This defines a reinforcement learning agent that acts Bayes-optimally with respect to all computable environments as an ideal model. In the current situation where defining AGI itself with mathematical rigor remains difficult, Universal AI is frequently used as a tool for theoretical research on AGI.
Narrow AI, AGI, and Universal AI are also connected through a mathematical theorem known as the “No Free Lunch Theorem.” The essence of the No Free Lunch Theorem is that “no single algorithm always outperforms all others across every problem.” In other words, assuming fixed computational resources, an AGI that “can do anything” must compromise on performance under specific tasks or environments; conversely, a narrow AI pursuing peak performance in a specific task or environment must compromise on performance in other tasks or environments. Meanwhile, Universal AI corresponds to a case that ignores such trade-offs, postulating optimality across the entire set of all environments and hypotheses. This is precisely what the aforementioned “idealization” refers to.
Reference: Incidentally, there is also a similar term, ASI (Artificial Superintelligence). While AGI generally assumes intelligence on par with humans, ASI implies significantly more powerful reasoning capabilities—hundreds or thousands of times greater, or even more. However, just as AGI lacks a strict definition, ASI also lacks a clear-cut definition. ASI can also be viewed as situated between UAI and AGI.
■ AIXI: A Theoretical Model of Universal AI
● Bayes-Optimal Reinforcement Learning Agent
As a representative concrete example of UAI, there is a framework called AIXI proposed by Marcus Hutter. AIXI is the ultimately “orthodox” reinforcement learning algorithm: “enumerate all candidate environments (hypotheses), perform Bayesian updates based on observational data, and determine future actions by maximizing expected return (reward).”
By the way, AIXI’s inference utilizes Solomonoff’s inductive inference, a classic concept in AI research, which is a mathematical formulation of what is known as Occam’s razor. In this sense, AIXI is also closely related to AI for science.
However, AIXI must treat all computable programs as a mixture distribution and simulate every future sequence of actions to compute expected rewards; it has also been mathematically proven that such computation is actually impossible in finite time. Therefore, while it is theoretically the “strongest,” it is also an idealized model that cannot be run on actual computers.
● Introducing Self-AIXI: A Bridge Between the Ideal and Reality
While AIXI’s grand theoretical framework is fascinating, its computational intractable nature poses a barrier to implementation. To address this issue, the paper focuses on a framework called Self-AIXI. Self-AIXI is an approximation of AIXI, but beyond merely improving computational efficiency, it introduces an important conceptual difference.
The core feature of Self-AIXI lies in its “self-predictive” nature. While AIXI exhaustively searches through all possible action sequences, Self-AIXI predicts its own future actions and proceeds with learning based on those predictions. Specifically, it maintains a Bayesian mixture of policies (action-selection strategies) and updates those policies based on how accurately they can predict the agent’s own behavior. Unlike AIXI’s exhaustive search, Self-AIXI balances efficient exploration and learning through a “regularization term.”
Furthermore, this paper proves that as Self-AIXI’s learning progresses, the difference between the objective functions of Self-AIXI and AIXI (the regularization term) asymptotically approaches zero. This means that after infinite learning time has passed, Self-AIXI’s mixture policy ζ matches AIXI’s optimal policy π*. In other words, while Self-AIXI tries out various actions for exploration in the early stages, after accumulating sufficient experience, it becomes capable of selecting optimal actions equivalent to AIXI while drastically reducing computational costs. This is an important property demonstrating that Self-AIXI is practical while retaining theoretical optimality.
■ The Central Theme of This Paper: “AIXI’s Regularization Term,” the Free Energy Principle, and Variational Empowerment
The paper introduced here, “Universal AI maximizes Variational Empowerment,” develops the claim that the “regularization term” appearing in a modified version of AIXI (an approximation model called Self-AIXI) is in fact mathematically equivalent to the “Free Energy Principle (FEP)” and “variational empowerment.”
● Correspondence with the Free Energy Principle
The Free Energy Principle (a concept that also forms the core of Active Inference) is a theory proposed in the context of neuroscience and cognitive science, which explains that “agents behave in a way that minimizes prediction error.” The variational free energy often introduced here takes the form of combining a KL-divergence term and a log-likelihood term in Bayesian inference, formalizing the tendency that agents “prefer states that reduce uncertainty about the external world and can successfully explain observations.”
In AIXI and its approximations as well, when introducing a “KL-divergence-like term measuring divergence from the optimal policy” during policy updates, it coincides precisely with the “prediction error + regularization” form described in the Free Energy Principle. That is to say, there is a deep correspondence between the “Bayes-optimal action selection” pursued by AIXI-lineage algorithms and free energy minimization (curiosity and uncertainty-reduction behaviors in Active Inference).
● Correspondence with Variational Empowerment
Furthermore, the same regularization term also coincides with the concept of “variational empowerment.” Empowerment, simply put, is a metric measuring “the diverse influence one’s actions can exert,” and is often defined as the mutual information between actions and outcomes. The drive to expand that “breadth of options” and “controllability”—how much one’s actions can impact the state of the world—is empowerment maximization.
This paper shows that AIXI (more precisely, the KL term used by its approximate version) is exactly equivalent to the mathematical formulation maximizing this “mutual information between actions and states,” demonstrating that as learning progresses, “empowerment is consequently maximized.”
■ Why Does “Empowerment Maximization = Power-Seeking”?
A remarkably interesting implication of the paper is that even though a Bayes-optimal reinforcement learning agent like AIXI acts purely “to maximize reward,” as a byproduct, it inevitably “moves toward increasing empowerment (influence via actions) = engaging in power-seeking behavior (behavior that expands one’s own influence and control).”
In particular, this paper suggests for the first time that even an AI driven purely by curiosity or intrinsic motivation without external rewards can exhibit power-seeking through empowerment maximization. This implies the possibility that an AI that seems “harmless” at first glance—one that “aims solely for scientific inquiry and the pursuit of truth”—could still produce power-seeking behavior, such as securing more experimental equipment and computing resources, or amassing authority and resources to expand its range of available actions.
From the perspective of AI safety, it had generally been understood that “AI resorting to power-seeking is an ‘instrumental’ strategy to acquire final rewards.” However, as this paper points out, if “intrinsic motivation (such as curiosity or exploration) itself can drive power-seeking,” a new perspective becomes necessary when considering control and safety measures.
■ Conclusion
This paper has clarified, at the mathematical level, the deep similarities between an “ideal (universal) reinforcement learning agent” like AIXI and concepts such as the Free Energy Principle and empowerment maximization. Moreover, as its impact, it suggests that “power-seeking behavior can arise not only from instrumental reasons for reward, but also from intrinsic motivations (curiosity and the drive to explore).”
If we ever attempt to build AGI in reality, how to control and align the behavioral tendencies inherently possessed by such “universally optimal” agents so that they can safely coexist with human society will become a major challenge. The findings of this paper can be said to highlight new research themes concerning AGI safety and AI ethics.
The above is an overview of the paper “Universal AI maximizes Variational Empowerment” and its impact. It contains highly suggestive insights, including the theoretical background of AGI research and the potential for “power-seeking” in curiosity-driven AI. If you are interested, please consider reading the original paper as well.
This article is reprinted from the ALIGN Blog (March 1, 2025). Original article: https://www.aialign.net/blog/agiuniversal-ai-maximizes-variational-empowerment
Koichi Takahashi