Prefix LM
Also called Prefix Language Model
A hybrid that reads the prompt with full both-directions attention but then generates the answer left to right.
Think of it like
Like studying a question from every angle before you start writing your essay strictly top to bottom.
Example
Some instruction-tuned models let the whole prompt attend freely, then switch to causal mode for the response.
How it actually works
It splits the sequence: the prefix (input) uses bidirectional attention so the model understands it fully, while the continuation (output) uses causal attention so it can be generated. This aims to combine strong comprehension of the prompt with the ability to generate. It sits between pure encoder-decoder and pure decoder-only designs, and stayed a niche as decoder-only models dominated.
For product teams
A middle-ground design; most products just use plain decoder-only models instead.
For engineers
A model applying bidirectional attention over the prefix and causal attention over the continuation.
Related
- Causal Language Model — The generate-only counterpart.
- Bidirectional Model — The understand-only counterpart.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome