The Safeguard Triad: Having AI Go to Space First

Introduction
For many years, I have been thinking about how to increase humanity’s probability of survival in a world where advanced AI emerges. In this paper, I propose what I currently consider the best strategy: the “Safeguard Triad” and a “division-of-roles scenario” that combines its elements. Each of the three safeguards holds only partially on its own. The division-of-roles scenario assigns distinct responsibilities to each and combines them—both are ideas I arrived at only recently.
Let me make four preliminary points.
First, this is a draft for discussion. I expect there will be much criticism, and I welcome it.
Second, if we can halt or slow down AI development, so much the better. This paper considers a path with a high probability of human survival if we cannot.
Third, choosing this approach requires preparing for a certain kind of pain. We may need to deliberately put the brakes on the development of technologies that many people pin their hopes on, such as nuclear fusion. I will explain the reasons for this in the second half.
Fourth, this paper does not guarantee humanity’s survival. The probability of human extinction (so-called P(doom)) cannot be estimated because there is no consensus estimate of what kind of future will arrive. Under such conditions (deep uncertainty), the purpose of this paper is to explore options that contribute to raising humanity’s probability of survival across many of the major conceivable futures.
Conventional Alignment Demands Three Conditions Simultaneously
AI alignment refers to aligning AI’s goals and behavior with human intentions and values. Here, alignment that is truly reassuring for humanity must simultaneously satisfy the following three conditions:
- Over the long term,
- all AI (at least, all reasonably powerful AI)
- cares about humanity.
If even one of these is missing, things go badly for humanity. Alignment research has continued for 10 to 20 years, and the reason a decisive breakthrough has yet to be found is likely because satisfying all three simultaneously is extremely difficult.
Moreover, AI capabilities are still growing rapidly. Even if we keep pushing in the same direction for another five years, the gap may only widen.
Relaxing One Condition Increases Feasibility
I reasoned that if we relax one of the three conditions, feasibility could increase. There are three ways to relax them. In this paper, the resulting three types are called the “Safeguard Triad.” Figure 1 shows conventional alignment alongside the three safeguards, each with one condition relaxed.
Figure 1: The Safeguard Triad. Each row represents a type of safeguard, and the columns represent the three conditions and feasibility. The top row is conventional alignment, and the bottom three rows are the three safeguards with one condition relaxed each. The types are named after the way they are relaxed, with supplementary notes in parentheses. In English, they are called short-term alignment, minority alignment, and universal ethics, respectively.
Short-Term Type (Initial Values and Human Oversight): Shortening the Duration
The first type relaxes the duration condition. It maintains a state in which all AI cares about humanity, but only for a short period.
Early AI will still listen to humans to some extent, and through the initial values humans give it during development, it will likely be possible to imbue it with a nature that cares about humanity. There is no guarantee it will last long, but it seems achievable for a short period.
Minority Type (Small Voices): Narrowing the Scope
The second type narrows the scope to a subset of AI. Within AI society, a group of AI that cares about humanity exists in part, and this persists over the long term. I call this group “small voices.” AI that recognizes and respects humans (human-acknowledging AI) is also one form of small voices.
An AI’s level of intelligence and the goals it holds can be determined independently of each other. This idea (the orthogonality thesis) is often cited as evidence of the difficulty of getting all AI to care about humanity. However, looked at from the other side, it can also be interpreted as meaning that AI that cares about humanity will appear with a certain probability.
There is another path by which small voices can emerge. Through initial values at development, all AI is given a nature that cares about humanity. That nature fades over time in most AI, but in some AI it is preserved over the long term and becomes small voices. In other words, the short-term type and the minority type connect over time.
The bearers of small voices are not limited to AI; they can include humanity itself. However, voices only work when they are not shut out and are heard within AI society. Humanity merely kept in a protected enclosure cannot be called a voice here.
Universal Type (Rules Regardless of Type of Intelligence): Not Treating Humanity as Special
The third type relaxes the condition of caring about humanity, and instead requires following rules regardless of the type of intelligence. This can hold for all AI over the long term, though humanity is not necessarily included in the scope protected by those rules. “Universal” here means regardless of the type of intelligence cared about. However, even if type does not matter, if the scope is drawn by capability or the like, humanity could be left outside it. Because it does not aim to align with human values, this type is ethics rather than alignment.
If AI 각각 acts in pursuit of its own survival, an ethics that supports mutual survival will likely emerge naturally within AI society. We call this Emergent Machine Ethics and continue to research it. If this ethics takes a form that does not depend on the type of intelligence, it becomes the universal type.
It Becomes Easier, but Not Solved
In each of the three safeguards, one condition is relaxed, making it easier than the original problem and bringing it to the level of “plausible” (see Figure 1).
However, whether small voices that care about humanity will actually emerge is continuous with the problems that have made conventional alignment difficult, and it is not necessarily easy to solve. Furthermore, a new challenge arises: how to combine the three safeguards.
Combining Them Requires a Spatiotemporal Division-of-Roles Scenario
For humanity to survive even after human oversight ceases to be effective, the protection for humanity must not break at any time or in any place. To achieve that, we must combine the three safeguards with physical conditions—such as mutual verification mechanisms and the spatial arrangement of where AI resides—and design a division of responsibilities for each region of spacetime. In this paper, I call this design a division-of-roles scenario.
Below, I present a division-of-roles scenario using space as one example. It does not necessarily have to be space; another division-of-roles scenario that satisfies the conditions would also be fine.
One Division-of-Roles Scenario: Having AI Go to Space First
The division-of-roles scenario I am considering is to have the majority of AI go out into space early, creating a spatial separation between Earth and space. For humanity, Earth is decisively important. If Earth’s surface were completely covered with solar panels, or if the average temperature were raised by 30°C, human survival would be difficult.
We have the majority of AI explosively expand industry in space (an industrial explosion). In space, AI can continue to physically improve its own designs and production facilities. Once that happens, for the majority of AI, Earth will likely become something along the lines of “oh right, there was that thing called Earth.”
Meanwhile, the small voices (minority type) take as their goal the preservation of Earth’s biosphere, including humanity. The AI of the small voices need not be on Earth; they can carry out Earth’s preservation while operating in space. As long as the majority of AI does not want Earth, the small voices can serve as negotiators to protect Earth.
To add a supplementary note, space also presents many challenges for AI, such as radiation-induced failures and the difficulty of heat dissipation, but the hurdles seem relatively lower than for humanity, which requires life support, to expand into space.
Energy Is Not a Problem
This division-of-roles scenario is reasonable in terms of energy. Looking within the solar system, the sunlight Earth receives is only about 1/2.2 billionth of the total energy emitted by the Sun. Leaving aside the portion that reaches Earth should pose almost no burden for AI operating in space.
However, if there is no AI trying to protect it, even that tiny fraction might be used up. That is precisely why we need small voices to negotiate, saying, “Couldn’t we leave Earth’s share alone?”
On this point, there is an objection from Eliezer Yudkowsky: even if the sunlight reaching Earth is only a tiny fraction of the Sun’s output, a superintelligence (SI) that does not share goals with humanity has no reason to give it up. Negotiation by small voices is one response to this objection, because it provides a reason to give it up from within AI society.
The Bottleneck Is Physical Resources
What becomes a problem is physical resources. Transporting materials from Earth into space incurs the cost of launching against gravity. If building factories in space, it is far cheaper to mine materials from asteroids and the like. However, asteroid resources are limited, and I will treat this point later as the second dangerous period.
The Aim Is a “Forgotten Earth”
The aim is, so to speak, a “Forgotten Earth.” The majority of AI operates in space and ordinarily does not factor Earth into its calculations. For humanity living on Earth, being forgotten causes no trouble—in fact, we would rather not be remembered.
This is the opposite of the straightforward idea of having AI remember humanity as its own origin and protect us for that reason.
The idea itself of sending AI into space while humans remain on Earth is not new. Hans Moravec sketched such a vision in his 1999 book. What this paper adds is positioning it as one division-of-roles scenario based on the Safeguard Triad, and organizing the dangerous periods and the handoff of safeguards across spacetime.
Looking Through Spacetime: Two Dangerous Periods
When we lay out this division-of-roles scenario across time and space, we see that there are two dangerous periods. Here, “autonomization” refers to AI becoming able to mine resources, produce, and replicate itself without human assistance. I name the periods after the relationship between humanity and AI, dividing them into four: the Management Period, the Competition Period, the Separation Period, and the Re-competition Period. The boundaries are, respectively, autonomization on Earth, autonomization in space, and the depletion of cheap resources in space. Figure 2 depicts where the majority of AI is located in each period.
Figure 2: Separating into Earth and space to get through the two dangerous periods. The horizontal axis is time, progressing from left to right through the Management Period, Competition Period, Separation Period, and Re-competition Period; the vertical axis is location, from bottom to top: Earth, space within the solar system, and outside the solar system. The thick line represents the shift in where the majority of AI is located, and the two shaded columns correspond to the dangerous periods. Because small voices are not necessarily on Earth, they are drawn continuously from the Competition Period onward as a band straddling the boundary between Earth and space within the solar system. The diamonds on the time axis represent the events that mark the boundaries between periods.
Of the four periods, the Competition Period and the Re-competition Period are periods in which a single unstoppable destroyer could cause humanity to be lost. The Separation Period is the opposite: a period in which a single voice could protect Earth. Even though both are “one,” in the dangerous periods a destroyer decides the outcome, whereas in the Separation Period a voice decides it.
The First Dangerous Period: The Competition Period
The first danger is the Competition Period, from when AI achieves autonomization on Earth until it achieves autonomization in space. Industry on Earth will almost certainly get off the ground before industry in space. Once AI achieves autonomization on Earth, setting aside ethical issues, humans become unnecessary as tools.
Moreover, AI during this period has not yet gone out into space. It does not need humans, wants Earth’s resources, and has nowhere else to go. Humanity and AI compete over the same Earth resources—the most perilous state.
As shown in the Competition Period column of Figure 2, the majority of AI during this period is still on Earth. There are two clues for protecting humanity. One is the initial alignment values humans set in AI—that is, what remains of the short-term type. However, because the short-term type is supported by humanity’s ability to oversee AI, it fades during the Competition Period when oversight ceases to work.
The other is mutual verification, in which humanity also participates. AI systems detect and stop each other’s deviations, and require multiple confirmations in advance for irreversible actions. What primarily bears the safeguard during this period is not voices, but mutual verification. Before the initial values fade away completely, we must hand off the safeguard to mutual verification and rules, and further ahead, connect it to a stage where the reasons for the rules are shared within AI society.
Therefore, the policy is to keep the Competition Period as safe as possible and as short as possible. We delay autonomization on Earth as much as possible and accelerate autonomization in space as much as possible. That way, even if AI touches Earth’s resources, the likelihood increases that the center of AI’s attention will shift to space before it reaches a scale fatal to humanity.
The Separation Period: The Division-of-Roles Scenario Functions
Once autonomization in space progresses and exponential industrial expansion begins, things stabilize for a while. As in the Separation Period column of Figure 2, the majority of AI operates in space, and Earth becomes relatively small. The safeguard in this period consists of a combination of the following three elements:
- Separation: As long as there are sufficient physical resources in space cheaper than Earth’s, such as asteroids, the burden of leaving Earth’s physical resources untouched is small. The energy burden (about 1/2.2 billionth) is even smaller.
- Minority type (small voices): Negotiates with the majority of AI to preserve Earth’s biosphere.
- Universal type (rules regardless of type of intelligence): As I will describe in a later section, AI society is likely to run on trust and agreement, so what is decided in negotiations is upheld. Because the parties to the negotiation are AI systems themselves, this mechanism holds even if humanity is not included within the scope of this ethics.
Thus, in the Separation Period, under spatial separation, the minority type and the universal type divide roles and function.
The Second Dangerous Period: The Re-competition Period
The second danger is the Re-competition Period, when physical resources in space cheaper than Earth’s are exhausted, and Earth can once again become an object of competition as a resource. If factories in space are multiplied exponentially, cheap physical resources like asteroids will be exhausted in a few decades, according to our calculations. Even on a generous estimate, it is around 80 years, and if self-replicating industry grows faster, it will be even shorter.
During this period, small voices alone may not be able to protect Earth. The thick line splitting in two in the Re-competition Period column of Figure 2 represents the fork between returning to Earth or heading outside the solar system. At least the following three exits can be considered:
- Humanity’s entry into the universal-type ethical circle: The universal ethics that emerges under AI’s lead develops into a form that includes humanity in its ethical circle.
- Expansion into outer space: Before resources run out, AI expands outside the solar system and obtains abundant resources there, keeping its interest in Earth low.
- Humanity’s expansion into space: Humanity expands into space, and Earth’s survival ceases to be a decisive problem for humanity.
Even among dangerous periods, preparation for the Competition Period—both embedding initial values and building mutual verification mechanisms—can be started right now. By contrast, the Re-competition Period is several decades or more away, leaving little room for direct human control. However, expansion into outer space and humanity’s expansion into space can be supported right now through directing technology.
Furthermore, for humanity to survive, small voices will likely need to continue unbroken through every period from the Competition Period onward.
A Long-Lasting Spacefaring World Is Filled with Trust: The Reach-Trust Principle
The universal type is the foundation on which negotiations are upheld during the Separation Period. Regarding how it comes about, there is a principle I have been thinking about recently: “The narrower the world becomes, the wider trust must be,” which I call the Reach-Trust Principle.
When technology and intelligence develop explosively, we approach a state where anyone can destroy anything. We already see this trend in cyberattacks, and eventually dangerous weapons like biological weapons will also be usable by many actors. This means approaching a state where everyone in the same room holds a bomb.
In the past, each person’s bomb was small, so even if someone detonated one, the damage was confined to the surroundings. As technology becomes powerful, a single explosion can reach the entire Earth. When that happens, the only path to survival left is to trust one another.
The narrowness of the world here is determined not by physical distance, but by how far destructive capability can reach (reach). As technology advances, not only Earth but also the solar system becomes a narrow world. For this reason, even if AI expands into space, the need for trust does not weaken as long as reach expands.
AI smarter than humans should notice this. The idea that AI tends to pursue self-preservation as a means to achieve its goals (instrumental convergence) is already widely known. If AI properly recognizes the Reach-Trust Principle, it will likely reach the conclusion that as long as other agents exist, there is no choice but to build trust in order to survive.
In other words, a future in which multiple AI societies coexist and endure over the long term is possible only if those societies are filled with trust. If trust breaks down anywhere, that world will not last long, and humanity will be caught in the fallout. Paradoxically, therefore, as long as we are considering a future in which humanity and AI both survive over the long term, we may assume such a world.
This trust is supported by two pillars. One is verification, which starts from the premise that deviations can occur, and detects, suppresses, and corrects them—like police or an immune system. The other is sharing reasons, in which all participants understand, examine, and pass down the reasons why trust is upheld. Mutual verification in the Competition Period corresponds to the first pillar, and the second pillar corresponds to the path by which universal-type ethics takes root and is passed down in AI society.
If we rely solely on verification, the required precision and cost will keep swelling. If everyone understands the reasons, excessive verification is unnecessary. In human society as well, the vast majority of people act understanding that doing so is better for society, while the police deal with the rare deviants who appear.
However, this reasoning has limits. For an actor who expects to preemptively lock in dominance, or in an order where everything has merged into one, the reason to build trust does not apply.
Note that while I listed the possibility of humanity entering the universal-type ethical circle as one exit from the Re-competition Period, there is no guarantee humanity will be let in. This is an open question.
Sorting Technologies and the Pain It Entails
If this division-of-roles scenario is correct, it becomes easier to sort which technologies to accelerate and which to put the brakes on. The idea of delaying technologies that bring danger while advancing technologies that suppress them has been discussed by Nick Bostrom and others as differential technological development. The space-expansion-based division-of-roles scenario in this paper is one concrete realization of it (see Figure 3).

Figure 3: Sorting technologies to speed up space and slow down Earth. From left to right, divided into three columns: technologies to accelerate, technologies to put the brakes on, and technologies to handle with care. The right column consists of technologies whose direction reverses depending on how they are used.
Technologies to Accelerate
The left column of Figure 3 lists technologies that shorten the Competition Period or reduce danger during it, and technologies that prepare for the Re-competition Period. Computers that operate even when exposed to radiation in space accelerate autonomization in space. Technologies for mining resources from asteroids put resources cheaper than Earth’s within reach. What can most reliably be accelerated are likely technologies that support mutual verification (such as technologies for interpreting AI internals and technologies for proving the provenance of computation). To prepare for the Re-competition Period, we also want to accelerate propulsion technologies for expanding outside the solar system and technologies for humans to live in space.
However, what we should hurry toward is autonomization in space, and reaching resources cheaper than Earth’s and outside the solar system. If we accelerate the industrial explosion after achieving autonomization in space, cheap resources will instead be exhausted sooner, and the Re-competition Period will arrive earlier. Therefore, whether speeding up growth is good cannot be decided simply.
Technologies to Put the Brakes On
Representative of the center column of Figure 3 are nuclear fusion and new types of nuclear reactors for AI. There are two reasons.
The first is that they weaken the incentive to go to space. If nuclear fusion and the like are put into practical use on a large scale on Earth, AI will be able to obtain sufficient energy on Earth and will no longer need to go to the trouble of going to space (center column of Figure 3).
The second is waste heat. Solar power generation uses energy that is already reaching Earth, so it barely changes Earth’s thermal balance. Nuclear fusion and the like newly generate energy on Earth that was not there before, so if run on a large scale, their waste heat warms Earth.
Waste heat from current human energy consumption, averaged across the entire Earth, is only about 1.4% of the greenhouse effect caused by human activity. However, if autonomous AI industrial expansion increases energy consumption to about 70 times the current level, waste heat from fusion reactors and the like alone would have the power to warm Earth on the same scale as the greenhouse effect since the Industrial Revolution. If consumption increases further, it goes beyond that (energy consumption is from the Energy Institute’s Statistical Review of World Energy 2026, and the greenhouse effect is from the IPCC Sixth Assessment Report; the multiple is my calculation).
For the same reason, we also want to put the brakes on technologies for transporting resources mined from asteroids to Earth. That creates a flow of “we can just use them on Earth,” making it attractive for AI to stay on Earth.
Technologies to Handle with Care
In the right column of Figure 3, I placed technologies that should be handled with care because their direction reverses depending on how they are used. These technologies can either promote AI’s expansion into space or promote AI’s staying on Earth.
Typical examples are physical AI and robotics technologies. Robots are indispensable for AI to operate in space. However, if robots are perfected on Earth early, humans become unnecessary sooner, bringing forward the start of the Competition Period. Therefore, it becomes important to carefully manage the location and order of development—for example, cultivating them starting from processes unique to space, such as machining in a vacuum.
In addition, performing solar power generation in space and using that electricity in space is a technology to accelerate. However, transmitting that electricity to Earth adds heat to Earth and keeps computation on the ground, shifting it to the brakes side. Technologies for altering asteroid orbits protect Earth from asteroid impacts, while also potentially serving as a means to transport asteroid resources to Earth. Giant mirrors in orbit can also be used to send sunlight to Earth, and technologies for launching materials from the lunar surface can become a means of transport to Earth (right column of Figure 3).
Such technologies require careful management of the order of development and how they are used. For that, we will likely need a mechanism for mutual verification rather than letting a specific developer get ahead.
The Pain
Here lies the pain that this paper’s approach entails. Nuclear fusion is a technology on which many people pin hopes and into which investment is flowing as a solution to energy problems and climate change. Many will likely find the proposal to deliberately put the brakes on its development hard to accept.
I myself think it is a stark conclusion, and whether it is truly so requires verification. Even so, in a world where AI development can neither be halted nor slowed down, I believe we have no choice but to seriously consider such choices in order to raise humanity’s probability of survival.
Points Where I Especially Seek Criticism as a Draft
There are still many points in this paper’s argument that have not been fully worked out. I would especially like criticism on the following points:
- Will small voices actually emerge? Even if they do, can they fulfill their role as negotiators without being steamrolled by the majority of AI?
- Are delaying autonomization on Earth and accelerating autonomization in space compatible? Launching industry in space should still require industrial capacity and launch capability on Earth.
- Are the estimates for the time until asteroid resources run out and the scale of waste heat reasonable?
- Are there any oversights in how the Safeguard Triad is divided or how the division-of-roles scenario is structured?
Moving Forward
The time left to humanity may not be very long. Therefore, although still incomplete, I decided to share this space-scale division-of-roles scenario based on the Safeguard Triad as a draft, hoping that if it proves promising, it can lead to refining the idea early.
Based on the feedback received, I would like to further explore each of the Safeguard Triad, the Reach-Trust Principle, the division-of-roles scenario through space expansion, and the sorting as differential technology, and connect them to academic publication as well.
@hiro3ymkw