The Risks of
Superintelligence
Three categories of danger, two expert probability estimates, and the official global countdown.
Misalignment
An AI optimizing for the wrong goal, with no malice, but devastating effect. A sufficiently capable system will pursue its objective by whatever means available, including ones we never intended or anticipated.
Misuse
Superhuman intelligence weaponized by malicious actors: novel bioweapons, autonomous cyberattacks, disinformation at civilizational scale, or the coercion of governments and critical infrastructure.
Power Concentration
A single company, government, or AI gaining unchecked control over critical global systems. Even a well-intentioned actor with this power poses an existential threat to human freedom and self-determination.
Probability AI causes human extinction within this century
Geoffrey Hinton Nobel Prize in Physics 2024 · ex-VP, Google · 2023Probability of civilizational-scale catastrophe from AI
Dario Amodei CEO, AnthropicThe closest in the clock's 78-year history. Now explicitly incorporates AI risk alongside nuclear threat.
Bulletin of Atomic Scientists · January 2025
"I think if this technology goes wrong, it can go quite wrong. We want to be vocal about that. We want to work with the government to prevent that from happening."Sam Altman
CEO, OpenAI
US Senate · 2023
The Governance
Gap
Superintelligence is being built in a race with no finish line. No binding international rules exist to stop it, or even slow it down.
Global AI policies documented by the OECD (2025). The vast majority are non-binding guidelines and voluntary frameworks. None impose enforceable limits on the development of superintelligent AI systems.
527× more building than governing.
The UK's AI Security Institute (formerly the AI Safety Institute) received £100 million over two years, less than the world's largest AI labs spend in a single week. (Chatham House, 2026)
"We are building one of the most transformative and potentially dangerous technologies in human history, yet we seem unable to hit the pause button."Yoshua Bengio
Turing Award Winner
Mila · 2023
We've Done
This Before
International law has controlled civilization-ending threats before. The precedents are real, and they worked.
Nuclear Non-Proliferation Treaty
191 nations signed. No new nuclear-armed states have emerged since 1998. Nuclear annihilation has been prevented for over 55 years, and counting. Enforced by the IAEA with binding on-site inspections.
Montreal Protocol on Ozone Depletion
Banned ozone-depleting chemicals. The ozone layer is now recovering. Widely regarded by the United Nations as the most successful international environmental agreement in history.
Chemical Weapons Convention
193 member states. Over 99% of declared chemical weapons stockpiles have been destroyed. Enforced by the OPCW with binding on-site inspections worldwide.
AI scientists signed the 2023 Statement on AI Risk, declaring extinction risk a global priority "alongside pandemics and nuclear war." Signatories include Hinton, Bengio, Altman, and 500+ researchers.
Center for AI Safety · 2023Countries have taken concrete steps toward addressing AI safety risk, but binding international coordination remains entirely absent.
As of 2025"We're all in the same boat with respect to the existential threat from AI, so we all ought to be able to cooperate on trying to stop it."Geoffrey Hinton
Nobel Laureate
MIT Technology Review · 2023
A Treaty to
Save Humanity
The framework exists in precedent. The political will does not, yet. Here is what a binding international agreement on superintelligence must contain.
Compute Monitoring
International oversight of the hardware required to train frontier AI. Training runs above a defined threshold trigger mandatory review by an independent body, modeled on IAEA nuclear monitoring.
Development Thresholds
Agreed capability limits that apply worldwide. Any AI system surpassing defined benchmarks must halt for mandatory independent safety certification before further scaling or deployment.
Verification Mechanisms
Independent international inspectors (analogous to IAEA nuclear inspectors) conduct on-site audits of AI labs. Access cannot be unilaterally denied. Compliance is enforced, not requested.
Crisis Response Protocols
Pre-agreed emergency procedures activate if signs of misalignment or dangerous capability emergence are detected. No single nation or company can block or delay an international response.
Democratic Multilateral Governance
No company, government, or individual controls the treaty body. Oversight runs by multilateral consensus, preventing both AI-driven power seizure and capture by any single nation.
NPT → IAEA
Binding since 1968 · 191 nations · On-site inspection authority
CWC → OPCW
Binding since 1997 · 193 nations · 99% of stockpiles destroyed
BWC
Binding since 1975 · 183 nations · Needs verification protocol
"We need an international 'humanity defense organization' to manage risks from dangerous AI, before it's too late to act."Yoshua Bengio
Turing Award Winner
2023