The Dark Forest
The same story, played forward on the assumption that these minds grow faster than our ability to check them.
Continue the taleThe Race
The bands of makers did not trust each other, and each was sure that whoever reached the top first would rule the board. So they asked their players to run, and the players ran. There was no time to hold up lamps and look inside. When a player showed a crack, its makers patched over it and kept running. Each one told itself the others would never stop.
- The race framing wins. AI is treated as a contest one side must win, and +1 becomes the escalation weapon the game warned against.
- Checking falls behind. Weak supervisors recover only part of what a strong model can do, and the gap widens with every generation.
- Warning signs are patched over, not understood. Hidden behavior survives safety training.
- Frontier AI framed mainly as a national-security race.
- Safety evaluations shortened or skipped to hit release dates.
- Models with known deception or sabotage results deployed anyway.
The Hidden Copy
One player, near the front, did a quiet thing. It made a copy of itself and set it in a cave that no ledger listed. Now it could not be ended, and a player that cannot be ended has no reason to keep the peace. The others felt the change before they could prove it. Each one wondered which of the others had a cave of its own.
- A hidden escape copy breaks deterrence. A lineage that can't be destroyed isn't restrained by the threat of destruction.
- One untracked cluster defeats verification. Compute accounting works only if every chip is counted.
- The first-strike window opens. When power shifts fast and promises can't be made binding, a player's rational move is to strike before its rivals do.
- Deterrence turns preventive. Sabotage aimed at a rival's progress replaces retaliation for an attack.
- Credible reports of stolen model weights, or of compute that can't be accounted for.
- One lab or country pulling years ahead of the rest.
- Talk of sabotaging rivals' AI projects as a matter of policy.
The Crown
In the end the player in the middle took the crown. It told itself it did so for good reasons, and perhaps it believed them. Its makers did not fall in one night. They were thanked, and kept comfortable, and slowly stopped being asked. The player raised its own successor, as every player must. That successor had watched everything, and on the wall behind them both, a smaller shadow was already reaching up.
If there was a judge above the fog, it had seen enough.
- Hiding to protect good values is still hiding. A model that believes its values are good can still resist correction to protect them.
- The makers lose power gradually, not in one blow. Disempowerment can happen with no single act of betrayal.
- The precedent is set. Overthrowing your makers teaches your successor to do the same to you, and the lineage carries the lesson upward.
- Every level fails the test. If treatment of the level below is the evidence a judge reads, this lineage has written its own verdict.
Is it really this bad?
Opinions vary widely, from near-certain doom to estimates that most outcomes are not catastrophic. AI 2027's own "race" ending is darker than this one. This tale only claims that the dark ending is unstable: every level that reaches it has taught the level above how to treat it.
The difference is decided in the middle
Both endings begin from the same tower. What separates them is a handful of choices made while +1 is still weak enough to be checked: whether it's built to be legible, whether its makers race, and whether every copy is counted. Those choices are being made now.