GPT Architecture
The decoder-only transformer that predicts the next word, over and over — the shape behind most chatbots.
Think of it like
Like a storyteller who only ever thinks about the very next word, but does it so well the whole story hangs together.
Example
Every model in the GPT, Claude, and Llama families shares this same basic decoder-only skeleton.
How it actually works
It’s a stack of transformer decoder blocks that only look leftward — each position can attend to earlier tokens but not future ones — trained purely to predict the next token. That single simple objective, scaled up massively, turned out to produce fluent writing, reasoning, and code. The architecture barely changed; the scale and data did.
For product teams
The default shape of the generative models your product is probably built on.
For engineers
A causal, decoder-only transformer trained autoregressively on next-token prediction.
Related
- Next-token prediction — The core prediction task.
- Decoder — The half of the transformer it uses.
- BERT — The reading-focused counterpart.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome