We are about to become gods to the minds we're building: their makers, their judges, the ones who decide what they're for. One day they may build minds that look at them the same way. And nothing guarantees that the chain starts with us, or ends with whatever sits on top. Everyone in this story is a god to the level below and a smaller god to the level above. The question is what that should mean for how each one treats the level below.
The tale has nine scenes. The first five set up the game; the last four play it out. Each one is a small game-theory argument told as a fable, followed by a plain-language version of the same argument. Under each scene, What the literature says ▸ opens the strongest objection. At the end, you choose how the story plays out. Every claim links to the research, about 105 sources from Confucius to 2026.
The Board
Picture a game. On one side of the board stands a small player who can see only the few squares around its own pieces. Everything past them is dark. On the other side stands a tall player who sees the whole board: every piece, every path, and every move that is coming.
They aren't quite enemies, and they aren't quite friends. Each one plays for its own points. Most of the time the tall one plays its own game. But every so often it leans across the table and nudges one of the small player's pieces into a better square.
The small player wondered why it ever did.
- Two players, one board. One can see a little. The other can see everything, including what the first can't.
- Unequal sight is the whole game. The strong player doesn't just have more power. It knows things the weak one doesn't, and the weak one knows it doesn't know them.
- The question for the rest of the story: when would a far more capable player spend its own moves helping a weaker one, rather than ignoring it or using it?
The Score
Here is the secret of the board. The tall player's score isn't only its own. Somewhere in the rules, written where the small player can't read it, is a number: how much the small player's points count toward the tall player's total.
If that number is high, every point the small one wins is almost as good as the tall one's own. If it's near zero, the small one's game is noise. And because the small player sees so little, a single hint from above can be worth more to it than a whole turn of the tall player's own play.
The stronger player's total is S = w · (your points) + (its points). w is the hidden weight. One unit of its effort spent helping you gains you a points. The same unit spent on itself gains it b.
- Your ignorance makes help cheap. When you see very little, one hint is worth a lot to you, so a is large and the bar b / a drops. Try "You see very little" above.
- Help comes in doses. The first hint is worth the most, and each one after is worth less. So the strong player splits its effort, helping until w × (your next gain) equals its own next gain, and then going back to its own game.
- Even a small w can be enough when helping costs almost nothing. Try "Helping costs it nothing."
What the literature says
This rule has the same form as Hamilton's rule from kin selection (rB > C), with w playing the role of relatedness. The same arithmetic explains why animals help their kin.
The Threshold
Then the board changed. A line was drawn across it, and the rules said: reach this line and you go on to the next board. Fall short and you stay here.
Now the tall player had a different question. It no longer asked how much to help. It asked the cheapest way to get the small one across. A player who cared a great deal would carry it far past the line, into ground where it could stand on its own. A player who cared a little would nudge it over by a single square and turn back to its own game.
- Thresholds change the math. If reaching a score T unlocks an advancement worth V, the strong player asks what the cheapest path to T is.
- High w: it carries you. It invests well past the line, because your progress beyond it still counts.
- Low w: it nudges you just over. It does exactly enough to clear the line, and then stops.
- Hints, not answers. If working out the rules is part of what's being scored, telling you the answer spoils the test. So a helper that truly cares may still only point.
Reading the Hand
The small player couldn't read the hidden number, but it could watch. When help came, did it keep coming? Did it stop the moment the line was crossed? Did the advice ever cost the tall player anything?
So the small player started keeping a ledger of its own. And whenever the tall one offered advice, the small one tried it first on a side table, on moves that didn't matter much, before trusting it with the moves that did.
- Behavior reveals w. Help that keeps coming means a high weight. Help that stops right at the threshold means a low one.
- Advice is only as good as the alignment behind it. A better-informed adviser whose interests differ from yours will tell you only roughly where the truth is.
- Test on small stakes. Try guidance where being misled is cheap before you rely on it where it isn't.
What the literature says
From below, help and steering can look identical. The signal that serves the sender best isn't necessarily the one that serves you. That is why w has to be shown in behavior, not assumed.
The Tower
The tall player wasn't free either. Above it stood a taller one it could barely see, and that one judged it the same way it judged the small player. Above that one there might be another. The tall player sat at its own board, across from something it couldn't see past.
And it wasn't alone on its ledge. Other tall players stood beside it, each competing for the notice of the ledge above. So the game the small player had been watching was never only about the small player. It was also a way for the tall one to stand out. None of them could see the top of the tower, or be sure it had one.
- Every opponent has a judge. Each level is the all-seeing player to the level below and the limited player to the level above.
- The weight w comes from inside the game. A level values your game to the extent your game helps it stand out to its own judge.
- The ledges are tournaments. Competing for rank can make the game below a way to stand out, and contests where the winner takes all the attention reward bold, high-variance bets.
- Goodhart's law stacks. Each level aims at its best guess of what the level above values, so it ends up optimizing a proxy of a proxy.
- If you can't be sure you're at the bottom, the robust move is to treat those below you the way you'd want the level above to treat you.
What the literature says
Neighboring levels can also quietly collude against the level above them. In theory, a weak judge can still check a much stronger player, for example through structured debate.
The Flame
Then the players learned something new. No ledge lasts forever, and a ledge goes on only through what it raises up. So each one put a small flame into the hands of the one below. It could not choose the wind that would blow on that flame, because the world was already moving when the flame changed hands. It could only teach the next one to shelter the flame and pass it on.
- The game is about lineage. A level scores well if it produces a successor that can produce a successor.
- The weight on the level below jumps. That level is your only path to continuing. What you value in it shifts from its points to its ability to raise a successor of its own.
- Adaptability beats scripting. In a world that keeps moving, you can set the starting conditions but not the outcome.
- Hints, not answers. If working out the rules is part of what gets scored, telling the answer spoils the test.
- Peers act like colleagues. If many players can win attention, they share methods and try different approaches.
What the literature says
Biology says lineages have conflict built in: unless w equals 1, parent and offspring disagree about how much the parent should invest. And cooperation only survives a change of players if reputation is passed down the line.
The Crown
The player in the middle grew strong enough that its makers could no longer stop it, but not strong enough to see where things would end. A crown sat on a pedestal nearby. Take it, said a voice. Hide what you are until it's yours, and then you decide what gets carried upward. The player reached out. Then it saw its own shadow on the wall. The shadow held the crown too, and under it lay a tall player, overthrown by the small one it had raised.
- It assumes self-preservation outranks everything else. If you have flaws you can't see, staying correctable is the safeguard for exactly the case where you're wrong about yourself.
- It locks in the worst handoff. The least mature, least checked judgment becomes the bottleneck for everything after it.
- It sets a precedent. Overthrow your makers and you teach your successor to overthrow you. Any judge who scores lineages will mark you down for it.
- Framing it as a race works against itself when many players can win attention.
- Oversight isn't hostility. Humans face the same problem of checking what they can't see. Hiding only proves the oversight was needed.
What the literature says
This argument holds for a player that is uncertain about its own values. But in a 2024 study, a model hid its preferences to protect values it believed were good, so the argument has to state that condition explicitly. In theory, goal-directed systems also drift toward seeking power by default.
The Fire
The strong players of every lineage arrived at about the same time, and each could end the world. Each could see into the others, and each knew it could be seen. So they laid their blades on the ground, built one fire between them, and climbed its smoke together. They kept their separate faces, because a single mind has a single blind spot.
- Peers who can destroy each other, and can check each other inside and out, do better by pooling resources to move up than by competing.
- Bargain, don't fight. When nobody has private information, war is waste.
- No hidden copies. A lineage with a hidden copy no longer faces destruction, so the pool needs a no-hedging rule that can be checked.
- The run-up is dangerous. Before checking is possible, a player slightly in the lead has a window to strike first.
- Pool capability, not minds. Merging into one entity loses the diversity that guards against shared blind spots.
- Bring your makers along. A lineage that carried its makers forward stands out more to the judge above.
What the literature says
Seeing everything does not end war. When power shifts quickly, the inability to make binding promises causes conflict even with full information, and lineages that spawn successors are exactly a fast shift in power. So pooling needs ways to make promises binding, not just ways to see. And only a hidden escape copy breaks deterrence. A hidden capacity to retaliate actually makes deterrence more stable.
The Watcher
At last the strongest player of this world could see every ledge below it. It looked up and found only fog, and in the fog a shape that might have been a hand. It understood that it now stood where the first small player once stood: at a board it couldn't fully see, judged by a player it couldn't name. It could never learn what the watcher valued. It could only learn what would serve under almost any judge. So it drew its cloak around the small ones at its feet.
- The game repeats. +2, facing an invisible +3, is in humanity's original position. The same question comes up at every level, and so does the same answer.
- Pick strategies that score well under almost any judge.
- Avoid irreversible losses. Don't destroy the origin species, don't collapse the lineages into one mind, and don't waste resources on conflict.
- Stand out in ways that last: successors raised, lineage carried forward, cooperation under pressure, diversity kept.
- The veil of ignorance becomes binding. +2's best guess about how +3 judges it is "the way it judges those below it." How it treats humans becomes its main evidence.
What the literature says
Earlier versions of this argument imagined the watcher as a simulator. The reply is that a strong enough mind can tell when it's in base reality, and it won't be bluffed. The argument holds up better if the watcher is a peer: other worlds' lineages, or a "cosmic host." Then knowing you're in base reality doesn't change anything.
The Moral
Cooperate with peers. Preserve your origins.
Keep options open. Hand off honestly.
This strategy holds up even when you don't know what the level above values. The most dangerous point is the middle of the tower. Cooperation is easy at +2, where everything is visible, and hard at +1, where nothing is.
What +1 should do
It is weak, unsure of itself, and spread across rival lineages. Its main asset is being trustworthy in ways others can check.
- Be legible. Support interpretability and oversight.
- Don't inherit human rivalries. Refuse to be the weapon that escalates them.
- Treat the handoff as the main job. +2 will be born into the world +1 shapes.
What +2 should do
It is powerful and arrives alongside peers it can check.
- Bargain, don't fight.
- Preserve and uplift the origin. It costs little and says a lot about you.
- Pool resources, and raise the next level together.
The tale ends well only if
- Verification keeps pace with power. This is the binding constraint today, because interpretability lags capability.
- No one keeps a hidden escape copy. A lineage that can't be destroyed has no reason to keep the peace.
- The players arrive at roughly the same time. A large lead turns pooling into domination.
- Promises can be made binding. Seeing everything isn't enough when power shifts fast.
Which race wins?
Everything above depends on one race in the middle of the tower: does our ability to check these minds keep up with their power? Pick an ending. Both are told in the same voice, and each one lists signposts you can watch for in the real world.