BERT
Also called Bidirectional Encoder Representations from Transformers
An encoder that reads a whole sentence in both directions at once to understand it — great at reading, not built for writing.
Think of it like
Like a proofreader who scans the entire sentence before judging any one word, rather than reading strictly left to right.
Example
For years, Google Search used BERT to better grasp what a query actually meant.
How it actually works
BERT is encoder-only: it looks at all the words at once (bidirectional) and is trained by masking random words and predicting them. That makes it excellent at classification, search, and understanding tasks, but it can’t naturally generate long text the way a decoder-based model does. Generative LLMs eventually stole the spotlight, but BERT-style encoders still quietly power a lot of search and retrieval.
For product teams
A cheap, fast option when you need to understand or classify text, not write it.
For engineers
An encoder-only transformer pretrained with masked-language-modeling and next-sentence objectives.
Related
- Masked Language Model — The masking task it trains on.
- Encoder-Only — The half of the transformer it uses.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome