Existential Risk
Also called X-Risk
The contested idea that sufficiently advanced AI could pose a threat to humanity’s survival or long-term potential.
Think of it like
The debate over nuclear weapons in the 1940s — a new technology powerful enough that some argued it could end civilizations, others that the fear was overblown.
Example
Researchers who take x-risk seriously argue for caution on training ever-larger systems whose goals we can’t yet verify or control.
How it actually works
The concern is roughly: if we build systems far more capable than us without reliably aligning their goals to ours, we might lose the ability to correct or stop them. Proponents treat it as a low-probability, high-stakes risk worth serious preparation. Skeptics argue it distracts from concrete present harms like bias and misuse, or that the scenarios rest on shaky assumptions. It’s one of the most genuinely divisive questions in the field, and reasonable experts land on both sides.
For product teams
A framing that shapes regulation and public trust, whether or not you find the scenarios convincing.
For engineers
The argument that misaligned highly-capable systems could become uncontrollable; contested on both probability and on whether it diverts effort from present harms.
Related
- AI Safety — The far-tail end of this field.
- Alignment — The technical problem underneath the fear.
- Misinformation — Debated against concrete present harms.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome