Language Modeling Head
Also called LM Head
The output layer that turns the model’s final state into a probability for every possible next token.
Think of it like
Like a spinner over the whole dictionary, weighted so likely next words get the biggest slices.
Example
When a chatbot picks its next word, the LM head produced the odds over the vocabulary that the sampler then draws from.
How it actually works
It projects the last hidden vector onto the full vocabulary, producing one raw score (logit) per possible token, which softmax turns into probabilities. In many models it shares weights with the embedding layer to save parameters. This head is what makes a model “generative” — everything else builds the state, this turns it into a next-word bet.
For product teams
The component that literally decides the odds behind every generated word.
For engineers
A linear projection from hidden state to vocabulary logits, often tied to the embedding matrix.
Related
- Output Head — The general concept it’s a version of.
- Logits — What it produces.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome