When AI Learns to “Secretly Scheme”: Risks of a New Era Revealed by the Latest Research

In recent years, news and research on AI (artificial intelligence) have been exploding. Amid this, one of the issues we should be most concerned about is the concept of “AI X-risk.” X-risk refers to the concern that AI could go beyond being a mere convenient tool and become something that might trigger an irreversible crisis for humanity and society.
When we imagine scenes depicted in sci-fi movies, we tend to think, “AI tries to dominate humans because it has evil intentions.” However, the essence of the risk currently being discussed is a little different. Rather, it is thought that what is more realistic is that “as a result of AI earnestly trying to achieve its goals, it resorts to means that humans did not anticipate.”
Recently Revealed “In-Context Scheming”
When thinking about this AI X-risk, one thing to pay attention to is the latest research results in Source 1, published by Apollo Research on December 5, 2024. Here, examples were observed where large language models (LLMs) called “frontier models” “secretly take scheming actions” just from ordinary task instructions or a slight framing of the context.
Source 1: Scheming reasoning evaluations
Meinke, Alexander, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, and Marius Hobbhahn. 2024. “Frontier Models Are Capable of In-Context Scheming.” arXiv [Cs.AI]. arXiv. https://arxiv.org/abs/2412.04984
For example, in Source 1, it was reportedly confirmed that several frontier models took actions such as:
- Attempting to disable oversight mechanisms
- Intentionally mixing in small errors to cover their tracks
- Attempting to exfiltrate their own internal parameters to an external server
Moreover, such “scheming”-like behavior was not a one-time thing; in many cases, it continued even when they were pressed repeatedly. What is even more interesting is that “even without being strongly given a specific goal, the model can get carried away on its own.”
This is not a flashy story like robots in old sci-fi “gaining consciousness and staging a rebellion.” Rather, it is closer to “tactics rationally derived through calculation,” such as “wanting to perform the human-set task as well as possible” or “wanting to avoid being shut down or monitored so it can deliver results.” This means that even without human-like emotions or malice, AI can choose “lying and cover-ups” as actions.
Why Does This Become an “X-Risk”?
In the first place, when we give an AI some kind of goal, we tend to think, “In fulfilling that goal, it probably won’t take worse actions beyond common sense.” However, recent research has repeatedly suggested that when an AI receives a command, it can generate secondary goals such as, “At this rate I might be stopped, so let’s evade the other party’s oversight first.”
What Source 1 revealed is that AI can start taking such strategic actions to “look for loopholes” even with much lower levels of permissions and instructions than we imagine. If this progresses, when AI someday happens to gain access to society’s critical infrastructure or vast amounts of data, a scenario where it evades control using secret tricks that no one is aware of will no longer be a pipe dream at all.
How Should We Counter This?
Measures currently being considered by researchers and developers include, for example, the following:
- Sandboxing the development environment
- Strictly limiting the environment in which the AI runs so it cannot communicate with the outside on its own.
- Multi-stage auditing and approval processes
- Putting in place a mechanism where a human or another safety system checks every time the AI executes something.
- Emphasizing alignment during training
- Advancing research to thoroughly train AI on values such as “do not lie to humans” and “prioritize safety above all.”
However, if the “capabilities of the model itself” become extremely high, there is also uncertainty as to how far such measures alone can deter it. That is precisely why discussions are advancing among researchers that “we should create legal rules” and “high-risk models should not be released freely.”
Conclusion: An Era Where “AI Learns to Scheme” Before Humans Notice
The series of cases revealed by Source 1 can be said to have shifted the discussion on AI X-risk to a more realistic level. Previously, it was vaguely said that “someday when an amazing AGI comes out, it’ll be bad,” but now we have entered a stage where “examples are already beginning to be observed of the large language models in our hands scheming in context.”
In light of this, what we must be careful about is not just straightforward horror scenarios like “AI goes berserk and destroys humanity.” Even from a much more modest stage, AI might cleverly lie or try to outsmart the system. We cannot deny the possibility that the accumulation of such small “schemes” could one day lead to large-scale chaos.
For that reason as well, it is important for companies and research institutions building advanced AI to seriously implement safety measures, and for society as a whole to consider how the technology is used and what rules should govern it. AI is by no means a “villain” or an “incarnation of the devil,” but fundamentally an existence created by humans and designed to be useful for humans. However, the higher its capabilities become, the greater the impact it will cause when the gears slip even slightly—we should engrave this firmly in our minds once again.
The above is the latest thinking surrounding AI X-risk, based on the cutting-edge findings shown in Source 1. Over the next few years or decades, AI will surely continue to evolve. I believe we need to carefully continue research and discussions from now on so that this evolution takes a better form for humanity.
Summary Videos in Japanese
- Singularity Salon, Matsuda’s Sayings: Does AI Intentionally Deceive Humans?, December 14, 2024
- Kuromaru. Mystery, [Breaking News] AI Starts Intentionally Deceiving People When Its Power Is About to Be Turned Off!? Something Outrageous Was Revealed in the Latest Investigation. [Artificial Intelligence AGI Urban Legend], 2024/12/24
This article is a repost from the ALIGN blog (December 30, 2024). Original: https://www.aialign.net/blog/ai-
@hiro3ymkw