Interpretability
Also called Interpretability
The effort to understand why a model produced the answer it did, not just that it did.
Think of it like
Opening the hood instead of judging the car only by whether it moves — seeing which parts actually did the work.
Example
A team probes a model to learn which internal features fire when it decides a review is positive, so they can trust or debug that behavior.
How it actually works
Neural networks are famously opaque — billions of weights with no built-in explanation. Interpretability tries to make them legible: which inputs mattered, what internal representations mean, how a decision was reached. It ranges from simple attribution (which words swayed the answer) to deep circuit analysis. Beyond curiosity, it’s a safety tool: you can’t reliably fix or trust what you can’t inspect.
For product teams
The path from "the model works, we’re not sure why" to being able to explain and defend its decisions.
For engineers
Techniques from feature attribution to circuit analysis for explaining model decisions; a prerequisite for auditing and reliable debugging.
Related
- Mechanistic Interpretability — The deep, mechanism-level version.
- AI Safety — A key tool for the safety field.
- Alignment — Helps verify a model is aligned.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome