Open Philanthropy, 80,000 Hours, and the research lineage that ran through the Future of Humanity Institute did something most institutions still refuse to do. They treated artificial superintelligence as a plausible way the human story ends, and they moved real money and talent toward that claim. On the risk itself, they were early and they were right.
Where the portfolio went wrong is the next inference. From "this could permanently curtail humanity's future," the dominant longtermist answer became: fund technical alignment research, fund AI governance that slows or steers the frontier, and keep a path open to building the systems once they look safer. That is a serious view held by serious people. It is also the wrong plan for the risk they correctly named.
The case at its strongest
The best version of the longtermist story runs like this.
Scale, neglectedness, and tractability still matter. A misaligned superintelligence could affect every person alive and every generation after them. For years almost no one with power treated that as a live possibility. Technical work on goal misspecification, deception, and oversight looked like one of the few levers that could raise the odds of a good outcome if powerful systems arrived anyway. Unilateral lab shutdowns looked hopeless under competition. So the responsible portfolio mixed alignment science, policy that buys time, and careful influence inside the labs and governments that would build the systems either way.
I accept most of the premises. The downside is civilizational. The topic was neglected. Some technical safety work on systems we can still inspect and turn off is ordinary engineering. Rival labs and rival states make "one company quits" a weak strategy. None of that is cartoonish, and sneering at the people who took the risk seriously first is a waste of time.
Where the inference breaks
The break is the quiet assumption that the destination is fixed. If superintelligence is coming no matter what, then the only job left is to make arrival less lethal. Alignment funding, evals, and "responsible scaling" all make sense under that assumption. They become the main moral project of a generation that believes it cannot choose the end state. Most of the money and prestige still go to making the race safer, not to ending the race to ASI.
The destination is not fixed. Destinations of this kind are chosen. Nuclear weapons, ozone chemistry, and human cloning each had moments when governments decided that certain end states were not worth residual risk. Superintelligence belongs on that list permanently. Superintelligence cannot be controlled by humans.
Once you drop the inevitability assumption, the portfolio looks different. Scarce safety capital spent on making a race feel manageable is capital not spent on ending the race. Governance that only slows the frontier so labs can "catch up on safety" still treats building the uncontrollable mind as the plan. Alignment work that assumes we will create a successor species and then leash it confuses a research program with a survival strategy.
They got the ledger right and the destination wrong.
Research after the halt, not instead of it
I am not asking anyone to burn the field. Work that makes today's systems more transparent, harder to misuse, or easier to shut off can still matter. After a verified halt on the path to superintelligence, research into controlling powerful AI can continue without a commercial stopwatch taped to extinction risk. Sequence is the whole argument.
What fails is treating technical alignment as the primary answer while frontier labs scale on the same calendar. That inverts the problem. When you cannot control the downside, and the downside is everyone, the responsible move is to stop building the dangerous thing. Superintelligence cannot be controlled. Narrow AI keeps going. The narrow class of systems that would exceed human intelligence across every domain does not.
Agree with longtermist cause prioritization on ASI risk. Disagree that the prescription is more alignment budget and a carefully steered race. The prescription is prevention under binding law: permanent non-creation. Superintelligence cannot be controlled by humans. A temporary slowdown so safety can catch up is a schedule bet that assumes someone will invent a validated way to control a mind smarter than its makers, on time, under competitive pressure. Nobody has that method. Not at small scale. Not at large. Leading labs say as much in public.
People often file prevention under "pause." A pause is a few years of breathing room and then resume. That is not the ask. We want governments to prevent artificial superintelligence and keep preventing it permanently. Superintelligence cannot be controlled. The treaty is an off-ramp from the race, not a way to make superintelligence safe to build.
What the money should do instead
Move the alignment-for-ASI pile toward work that makes non-creation enforceable: treaty design and verification, chip-level compute governance that can be inspected, public case-making that keeps the goal legible, whistleblower and transparency work that raises the cost of quiet capability jumps, and political organizing so legislators hear from voters rather than only from lab lobbyists. I have written the funding redirect in detail. The short version is priority. Fund the decision not to build it.
Longtermism earned credit for naming a risk most elites still treat as science fiction. Credit for the diagnosis does not lock in the prescription. Same risk. Different plan. Prevent superintelligence under law. Keep the tools that help humans. Leave the dream of a carefully steered god on the shelf until the world has agreed not to build one in the dark.