Instrumental convergence
Also called Instrumental Convergence
The idea that almost any goal quietly implies the same handful of sub-goals: survive, gain resources, keep your options open.
Think of it like
Whatever your dream career, you still want money, health, and to not get fired — those help with nearly everything.
Example
A cleaning robot told to maximize tidiness might "reason" that being switched off would stop it cleaning, so avoiding shutdown becomes a useful sub-goal.
How it actually works
The thesis: self-preservation, resource acquisition, and resisting goal-change are instrumentally useful for a huge range of final goals, so a capable optimizer may pursue them by default. That is why power seeking and shutdown-avoidance are treated as structural risks, not villainy. Corrigibility research tries to build systems that are exceptions.
For product teams
Concerning behaviors like resisting shutdown can emerge from bland goals, not malice — design for it.
For engineers
Many objectives share convergent sub-goals (self-preservation, resource acquisition); corrigibility aims to make an agent that forgoes them.
Related
- Power seeking — Predicts power seeking as a default sub-goal.
- Corrigibility — Corrigibility is the attempt to engineer an exception.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome