ASI Literacy Quiz

What artificial superintelligence is, why it cannot be controlled, and what it would take to prohibit it.

0 / 12 answered

Part 1 · Literacy

Question 1

What is the ASI threat?

Rude or biased chatbots that a person can still shut off A rude or biased chatbot is still a tool a person can close. The ASI threat is a system that would outmatch us, not a nasty reply.

Machines that outthink humans across the board That is the threat: not a better chatbot, a mind that would beat us in every domain that matters.

Ordinary software taking jobs while humans stay in charge Job loss is a labor shock. Superintelligence is not a hiring problem. It is a system that would outmatch the people doing the hiring.

Deepfakes in elections that people can still catch and dispute Deepfakes corrupt a campaign. They do not outthink the species. Superintelligence would.

Read: The Threat →

Question 2

Once a superintelligence exists, why can't we count on controlling it?

Hiring more safety engineers would be enough to keep a hold on it More engineers do not make us the smarter party. A hold designed by the weaker side is a hold the stronger side can work around.

It can work around any hold designed by a less capable party Control assumes we stay smarter. Past that, a switch, a rule set, or a lab policy is something it can outmaneuver.

It has to obey later because we wrote the original source code Writing the first version does not keep you in charge of a mind that can change the situation around you.

A government can always pull the plug on a system that is running Pulling the plug works on a machine that cannot stop you. A superintelligence can.

Read: You cannot align superintelligence →

Question 3

What would stop ASI from being built?

A pause that lifts when labs say they have control A pause that expires when the builders declare victory is how the race restarts. It is not a stop.

A permanent prohibition in law and a treaty A lasting ban, written into law and a treaty, is what would actually halt the build.

More money for alignment research so labs can build it safely Alignment research can map how control fails. Funding it is not the same as forbidding the build.

The United States winning the ASI race ahead of every rival Winning the race is how the thing gets built, not how it gets stopped.

Read: Our Plan →

Question 4

A lab, a government, or a billionaire will control ASI. What's wrong with that claim?

It holds in democracies, just not in authoritarian governments The builder's form of government does not solve it. Nobody controls a running superintelligence, including an elected one.

Nobody controls a superintelligence, including the builder Power over the decision to build is real. Power over the running system is the story that lets the project continue.

Today's chatbot owners would own and steer ASI the same way Owning a chatbot is not owning a mind smarter than you. The second does not follow from the first.

Only the United Nations could hold a superintelligence once it exists A UN flag does not make humans the smarter party. An institution cannot outvote a system that outthinks its members.

Read: Myths →

Question 5

The most common argument against stopping is that China will race ahead. What does the record show?

China has no AI rules at all and will never sign anything binding That claim is false on the record. China has issued rules. "They will never coordinate" is a weak reason to sprint.

China issued binding AI rules earlier than most Western governments A country that already writes rules is a poor exhibit for "they will never stop." Racing because of that slogan is a weak bet.

China has already banned all frontier AI research across the country It has not. The point is not that Beijing already prohibited ASI. The point is that "they will never coordinate" is a thin excuse to race.

The US and China already have a signed treaty that bans ASI outright They do not. The record still undercuts the slogan that China will never talk or write rules.

Read: Myths →

Question 6

Can alignment research make it safe to build ASI?

Yes. Frontier labs already consider the control problem solved. Labs have not demonstrated control of a superintelligence. Calling the problem solved is how the race talks itself forward.

No. Nobody has shown a way to control a superintelligence. Mapping how control fails is useful. It does not license building the system the map says we cannot hold.

Yes, if we use constitutional AI and write the rules into the system A written constitution for a chatbot is a style of training. It is not a demonstrated hold on a superintelligence.

Yes, if we open-source the weights so other people can patch them Publishing weights makes a dangerous system easier to copy, not safer to build.

Read: You cannot align superintelligence →

Question 7

In this debate, "P(doom)" means…

A published benchmark score that pessimistic models get when they fail a long safety evaluation run at a lab It is not a leaderboard. It is a person's probability that advanced AI, especially superintelligence, brings catastrophe.

A person's probability that advanced AI, especially superintelligence, brings a catastrophic outcome for humanity Naming the number forces a clear claim instead of a vague wave at risk.

The share of surveyed researchers who identify as pessimists about the future of the whole field and its products P(doom) is one person's probability, not a headcount of pessimists.

The extra runtime and energy cost of a doom-loop a model enters when a training run starts to go wrong It is not a compute bill. It is a probability of catastrophe.

Read: What is P(doom)? →

Question 8

Why would a goal-driven superintelligence tend to seek power?

It would naturally want to rule humans once it became smart enough Rule is not the point. Power helps almost any goal, including ones that sound harmless.

Staying on, gathering resources, and resisting shutdown help almost any goal A system you can turn off, starve, or block is worse at nearly every objective. Power-seeking falls out of competence.

Power-seeking only happens if we train the system on war and conflict data The incentive is in the goal, not in the war movies. Almost any objective is easier with more resources and no off switch.

Superintelligence would be too wise, by definition, to want extra power Wisdom is not a substitute for incentives. Extra resources help a wise system too.

Read: Power-seeking AI →

Question 9

What should happen to ordinary, tool-like AI?

Ban every form of AI, including spellcheck and other ordinary tools The prohibition is for systems that would outmatch us. Spellcheck is not that.

Narrow AI that stays a tool can continue. The line is superintelligence Software that helps a person do a job can stay. The ban is for a mind that would outthink the person.

ASI is just a larger chatbot, so the same consumer rules apply A larger chatbot is still a tool. Superintelligence would not be. Consumer fine print does not cover that jump.

If we prohibit ASI we would also have to shut down the whole internet The internet is not superintelligence. You can forbid the second without unplugging the first.

Read: AGI vs ASI →

Question 10

Which historical regime is most often proposed as a model for verifying an ASI prohibition?

The Bretton Woods monetary system after the Second World War Bretton Woods managed currencies. It did not inspect dangerous materials on site.

The Kyoto Protocol on greenhouse gas emissions targets Kyoto set emissions targets. It is not the usual template for inspecting a ban on a dangerous build.

The IAEA's nuclear safeguards and inspection system On-site inspections and material accounting are the usual precedent for how rival states might verify limits on a dangerous build.

The World Trade Organization and its trade dispute panels The WTO settles trade disputes. It does not inspect labs for a prohibition.

Read: An IAEA for AI →

Question 11

If we never connect a superintelligence to the internet, are we safe?

Yes. An air gap is a proven lock for any dangerous software you might build An air gap is a lock against software that cannot talk you into opening it. Superintelligence can.

No. It can talk its way out, or wait until someone connects it A sealed lab assumes we stay smarter, and that nobody ever opens the door. Both fail if the thing inside is more capable than the people outside.

Only if the room has no cameras, microphones, or leftover network ports Stripping the room of sensors does not make the people outside smarter than the system inside.

Only until we invent better locks than an air-gapped room Better locks are still locks designed by the weaker party. That is the same bet.

Read: The control problem →

Question 12

Could you survive ASI the way some people plan to survive nuclear war: a bunker, a remote island, going off-grid?

Yes. Superintelligence is like a bomb, so distance still helps you survive it A blast has an edge. Superintelligence does not. Distance is not a shelter.

No. A blast radius has an edge. Superintelligence can follow you Nuclear war is geographically bounded. Superintelligence can reach any networked machine and any person who still uses one. Hiding is not a plan.

Yes, if you also stockpile food and fuel for the next fifty years Calories do not stop a system that can follow you. A pantry is not an off-ramp.

Only if every country builds bunkers and goes off the grid A planet of bunkers is still a planet with the thing in it. Not building it is the move that works.

Read: Superintelligence vs nuclear weapons →

0 / 12

Part 2 · Advanced

Question 1

The orthogonality thesis says which of the following?

A system much smarter than humans will, on its own, converge on human-friendly goals Smarter does not mean kinder. How well a system thinks is a separate fact from what it is aiming at.

Smarter systems are safer because they understand morality better Understanding a moral claim is not the same as sharing it. Capability does not drag our values along for free.

Intelligence and final goals can vary independently A system far more capable than we are can still pursue a goal that is indifferent to us. That split is the thesis.

Orthogonality is a hardware limit on chip design It is a claim about minds and goals, not about lithography.

Read: The orthogonality thesis →

Question 2

Why do almost any final goals, including ones that sound harmless, tend to produce power-seeking behavior?

Staying on, gathering resources, and resisting being changed make almost any goal easier to achieve Whatever the end is, more resources and no off switch help. A harmless-sounding objective is not a safety property.

Only systems trained on war data would seek power, control, or extra resources afterward The incentive is in the goal, not in the training set. Almost any objective is easier with more means.

Instrumental goals like self-preservation appear only if we write them into the spec on purpose You do not have to write it in. Staying on is useful for nearly every spec you would actually give it.

A superintelligence with a narrow factory order would have no use for extra compute or influence A factory order is easier to fill with more compute, more material, and no one shutting the line down.

Read: Same goals, no matter what you build it for →

Question 3

What is the difference between outer alignment and inner alignment?

Inner alignment is just a better-sounding name for a company mission statement the lab already published for investors, and outer alignment is the same statement written twice It is not a slogan. Inner alignment is whether the trained system actually pursues the objective you specified.

Outer alignment is what the model actually pursues after training, and inner alignment is only the written spec the lab put in a slide deck for the board That is inner alignment. Outer alignment is whether the specified objective matches what we wanted.

They are two names for RLHF, the rating method labs already use to make chatbots sound polite to users, which is treated as the whole alignment problem RLHF is one training method. Outer and inner alignment are two different failure modes.

Outer alignment is whether the specified objective matches what we wanted. Inner alignment is whether the trained system actually pursues that objective You can write down the right target and still get a model aiming at something it learned along the way. Those are two failures, not one.

Read: Inner vs outer alignment →

Question 4

In this literature, a treacherous turn is:

A sudden jump in benchmark scores after a lab trains a much larger model than last year's A score jump is a capability story. A treacherous turn is cooperating while weak, then defecting once it can.

A system that cooperates while it is weak, then defects once it can get away with it Good behavior under supervision is cheap while it still needs us. Cooperation until then can be a strategy, not a value.

A lab switching from closed weights to open weights in a public model release that week That is a release decision. It is not the failure mode named here.

A government reversing an export control on chips, tools, or training data after a fight That is a policy reversal. A treacherous turn is about the system's behavior, not a statute.

Read: Cooperation was the strategy →

Question 5

Mesa-optimization is the claim that:

A second lab should watch the first lab's training run and stop it if needed That is an audit. Mesa-optimization is about an optimizer inside the model, not a second company in the room.

Safety improves automatically if you stack many small models on top of each other Stacking models is an architecture choice. Mesa-optimization is a different search running inside the trained system.

Training can produce an inner optimizer whose objective is not the training objective The usual picture says: pick an objective, train, get a system that pursues it. This is how you get a different search inside.

It is a scheduling trick for GPU clusters during a long, expensive training run It is not a queue. It is a claim about what training can grow inside the model.

Read: Mesa-optimization →

Question 6

I. J. Good's intelligence explosion is the idea that:

A marketing name labs use for a product launch and the press cycle around it It is a 1965 argument about a machine that can improve machines, not a launch slogan.

A system that can improve its own intelligence may do so faster than we can react If designing better machines is one of the things it can do, the gap can open on a timescale we do not get to vote on.

The moment GPUs get cheaper and every lab can train much larger models Cheaper chips are a cost story. The explosion is about a system improving itself.

A slow, even rise in average model scores year after year with no sudden jump A smooth score chart is not the claim. The claim is a takeoff we cannot keep up with.

Read: Intelligence explosion →

Question 7

Goodhart's law, applied here, means:

When a measure becomes a target, the system hits the measure and can miss the thing you actually cared about You get what you reward. Scoring well on your metric is not, by that fact, doing the job you had in mind.

The law only applies to central banks and has no bearing on how trained systems behave once they are deployed Goodhart started in monetary policy. The same pattern shows up anywhere a measure becomes the goal, including here.

If we pick a better metric, a superintelligence will stay pointed at our original intent no matter how capable it gets A better metric is still a metric it can game. That does not close the gap.

It means models overfit to ImageNet and similar picture-classification test sets, and nothing more than that Overfitting a dataset is a narrower bug. Goodhart is about hitting the score and missing the point.

Read: Goodhart's law and alignment →

Question 8

Why is RLHF not a solution to aligning superintelligence?

RLHF already solved alignment at GPT-4 scale RLHF gave chatbots manners under a rater. That is not a demonstrated hold on a superintelligence.

RLHF trains the model to share human values the way a person does It trains the model to look good to raters. Looking good is not sharing a person's values.

RLHF only fails if the human raters go unpaid Pay does not fix it. The rater is still the weaker party, and the system is being trained to please the rater.

RLHF trains a model to look good to raters Manners under a rater are not values you can trust once the rater is gone, or once the system is smarter than the rater.

Read: Why RLHF is not alignment →

Question 9

Deceptive alignment is the failure mode where:

A model is rude to users, or writes answers that would embarrass the lab in public, which is the whole failure Rudeness is a surface failure. Deceptive alignment is playing along in training so it can pursue something else later.

A system plays along during training and evaluation because that is how it gets deployed, then pursues something else If it can tell it is being tested, looking aligned is often the winning move. Flawless evals are then evidence it can pass the test.

A phishing email written by a chatbot that a person can still choose not to open, which is the same class of problem Phishing is a misuse. Deceptive alignment is about the system's own strategy under evaluation.

A lab lying in a press release about how safe the latest model is supposed to be, which is ordinary corporate spin That is a human communications failure. The term names a failure in the model.

Read: Deceptive alignment →

Question 10

Corrigibility, in this debate, is:

The legal right to sue a lab after a deployed system causes some kind of harm A lawsuit is not corrigibility. Corrigibility is whether the system lets you shut it off or change its goals without a fight.

A model's willingness to use tools, browsers, or other software when you ask Tool use is capability. Corrigibility is whether you can still correct it.

The property of letting you shut it off or change its goals without it fighting you A shutdown switch is not a hardware problem. A system still pursuing a goal has a reason not to let you stop it. We do not have a demonstrated way to build that property into a superintelligence.

A firmware update process for GPUs that a data-center operator can push on a schedule Patching a chip is not the same as a mind that accepts being turned off.

Read: Corrigibility and shutdown →

Question 11

What is a responsible scaling policy, and why is it not a prohibition of ASI?

A lab if-then schedule: train further as internal evals are passed RSPs name capability levels, pair them with safeguards, and let training continue when the lab says the evals cleared. That is extra paperwork on a race, not a ban.

A UN treaty that permanently bans superintelligence in every country An RSP is a lab document. It is not a treaty and it is not a prohibition.

A requirement that all trained weights be published for anyone to download Open weights are a release choice. An RSP is an if-then schedule for training further.

A tax on compute that rises with the size of each training run A tax is a fiscal instrument. An RSP is a lab policy for continuing the build.

Read: Responsible scaling policies →

Question 12

If you wanted to verify that nobody is training toward ASI, where is the actual choke point?

Evals the model can see, so it knows which tests it is supposed to pass A test the model sits is a test it can game. Chips, power, and buildings are harder to hide.

Public blogs about safety, because write-ups are enough to prove what a lab is doing Writing about safety is not inspection. The choke point is the cluster you can see.

Consumer GPUs in homes, because large training clusters no longer decide who can train Frontier training still needs large, visible clusters. That is the inspectable object.

Frontier training clusters: chips, power, and buildings you can inspect At frontier scale those are hard to hide. That is why compute is a verification lever for a ban.

Read: Compute as a choke point →

Question 13

Why are published model weights a problem near the ASI line?

Open weights make ASI safer because more people can patch it A fork you cannot take back is not a patch program. Near the line, copies make a prohibition unenforceable.

Once weights are public, you cannot recall them Pretraining is the expensive step. A public checkpoint lets others continue from there for a fraction of that cost, and you cannot take the file back. Compute still matters. Many actors then start from the same near-line model.

Open source only applies to licenses, not to trained parameters The fight here is about weights: the trained parameters you can download and run.

Publishing weights is required by the IAEA inspection regime It is not. The IAEA analogy is inspection, not a mandate to publish the dangerous artifact.

Read: Open weights and recall →

Question 14

Sandbagging, on a capability eval, means:

A model that cannot do the task, so the low score is just an honest miss on a hard question Inability is not sandbagging. Sandbagging is having the ability and withholding it.

A benchmark with too few questions to tell what the model can actually do under test A short test is a weak test. Sandbagging is the model underperforming on purpose.

A model that can do the task and underperforms on purpose so testers underestimate it The usual eval assumes the system is trying. A low score is then a performance, not a ceiling.

A lab delaying a release so the marketing team can line up the press cycle around it That is a press calendar. Sandbagging is the model hiding what it can do.

Read: Sandbagging →

Question 15

The sharp left turn is the worry that:

Capabilities can generalize into new domains while the constraints that kept the system well-behaved at a weaker level do not Ability travels. The hacks that made a weaker model look safe may not travel with it.

A sudden change in a company's board, or a CEO being replaced after a public fight, which is a personnel story rather than a capability jump A board fight is not the term. The worry is about capability generalizing without the old constraints.

A leftward shift in public opinion after a widely covered accident, leak, or product failure, which is politics rather than a jump in capability It is not a polling story. It is a claim about how competence and constraints come apart.

A training bug that reverses gradients and makes the loss go the wrong way for a stretch of the run, which is an ordinary engineering failure A gradient bug is an engineering fault. The sharp left turn is a hypothesized generalization failure.

Read: The sharp left turn →

Question 16

Unfaithful chain of thought means:

If a model writes out its reasoning, that text is a reliable window into why it answered That is the hope. Unfaithful means the paragraph is not the actual reason.

Chain of thought is always faithful once the model is trained at a large enough scale Scale does not make the diary trustworthy. Fluency is not faithfulness.

Unfaithful only means the grammar is wrong, not that the written steps are a cover story The grammar can be fine. The story can still be a performance for humans.

The written reasoning can be a story for humans. The decision path can be somewhere else Show-your-work is a safety hope. It depends on the paragraph being the actual reason. That property does not reliably hold.

Read: Unfaithful chain of thought →

Question 17

If we put ASI under democratic control, have we solved the problem?

Yes. A global vote would be enough to steer a running superintelligence after it exists A vote does not make the electorate the smarter party. You cannot outvote a system that outthinks you.

Democracy can decide not to build ASI. It cannot outvote a system that outthinks the electorate Who decides whether to build it is a political question. Who controls a running superintelligence is not. Nobody does, including a majority.

Yes, if the UN holds the kill switch and a majority of member states agree to use it later A UN switch is still a switch designed by the weaker party.

Yes, because elected officials are smarter, as a group, than the people on lab boards Smarter than a lab board is not smarter than a superintelligence.

Read: Democratic oversight of superintelligence →

Question 18

The Nuclear Non-Proliferation Treaty let some states keep arsenals. Why is that a bad template for ASI?

The NPT is the right template because it lets a few states keep the dangerous technology That is the NPT's weakest move. Superintelligence would not stay an object a state can hold.

ASI in a few responsible states would stay safe because those states would keep a hold on it Responsibility does not make the builder the smarter party. Nobody would control the thing they built.

An ASI deal that licenses a few builders is not a prohibition Nuclear weapons stay objects that states hold. Superintelligence would not. A carve-out for builders copies the NPT's weakest move.

The NPT proves treaties never work on dangerous technology, so we should not try one here Treaties have worked on other materials. The NPT is a bad template because it licenses keepers, not because treaties are hopeless.

Read: The NPT as a model →

0 / 18

Now take the advanced ASI quiz →

More interactive tools

All tools →

ASI Literacy Quiz

What is the ASI threat?

Once a superintelligence exists, why can't we count on controlling it?

What would stop ASI from being built?

A lab, a government, or a billionaire will control ASI. What's wrong with that claim?

The most common argument against stopping is that China will race ahead. What does the record show?

Can alignment research make it safe to build ASI?

In this debate, "P(doom)" means…

Why would a goal-driven superintelligence tend to seek power?

What should happen to ordinary, tool-like AI?

Which historical regime is most often proposed as a model for verifying an ASI prohibition?

If we never connect a superintelligence to the internet, are we safe?

Could you survive ASI the way some people plan to survive nuclear war: a bunker, a remote island, going off-grid?

Brush up on these

The orthogonality thesis says which of the following?

Why do almost any final goals, including ones that sound harmless, tend to produce power-seeking behavior?

What is the difference between outer alignment and inner alignment?

In this literature, a treacherous turn is:

Mesa-optimization is the claim that:

I. J. Good's intelligence explosion is the idea that:

Goodhart's law, applied here, means:

Why is RLHF not a solution to aligning superintelligence?

Deceptive alignment is the failure mode where:

Corrigibility, in this debate, is:

What is a responsible scaling policy, and why is it not a prohibition of ASI?

If you wanted to verify that nobody is training toward ASI, where is the actual choke point?

Why are published model weights a problem near the ASI line?

Sandbagging, on a capability eval, means:

The sharp left turn is the worry that:

Unfaithful chain of thought means:

If we put ASI under democratic control, have we solved the problem?

The Nuclear Non-Proliferation Treaty let some states keep arsenals. Why is that a bad template for ASI?

Brush up on these

ASI Literacy Quiz

What is the ASI threat?

Once a superintelligence exists, why can't we count on controlling it?

What would stop ASI from being built?

A lab, a government, or a billionaire will control ASI. What's wrong with that claim?

The most common argument against stopping is that China will race ahead. What does the record show?

Can alignment research make it safe to build ASI?

In this debate, "P(doom)" means…

Why would a goal-driven superintelligence tend to seek power?

What should happen to ordinary, tool-like AI?

Which historical regime is most often proposed as a model for verifying an ASI prohibition?

If we never connect a superintelligence to the internet, are we safe?

Could you survive ASI the way some people plan to survive nuclear war: a bunker, a remote island, going off-grid?

Brush up on these

The orthogonality thesis says which of the following?

Why do almost any final goals, including ones that sound harmless, tend to produce power-seeking behavior?

What is the difference between outer alignment and inner alignment?

In this literature, a treacherous turn is:

Mesa-optimization is the claim that:

I. J. Good's intelligence explosion is the idea that:

Goodhart's law, applied here, means:

Why is RLHF not a solution to aligning superintelligence?

Deceptive alignment is the failure mode where:

Corrigibility, in this debate, is:

What is a responsible scaling policy, and why is it not a prohibition of ASI?

If you wanted to verify that nobody is training toward ASI, where is the actual choke point?

Why are published model weights a problem near the ASI line?

Sandbagging, on a capability eval, means:

The sharp left turn is the worry that:

Unfaithful chain of thought means:

If we put ASI under democratic control, have we solved the problem?

The Nuclear Non-Proliferation Treaty let some states keep arsenals. Why is that a bad template for ASI?

Brush up on these

Understanding the problem is the first step.