Embedding Layer
The first layer that turns each token ID into a vector the rest of the network can actually work with.
Think of it like
Like a coat check that swaps your ticket number for the actual coat — the number meant nothing until it fetched the real thing.
Example
When token #4021 for “dog” enters a model, the embedding layer looks up its learned 4,096-number vector.
How it actually works
It’s essentially a big lookup table: one learned vector per vocabulary entry, trained alongside everything else. Raw token IDs are meaningless integers; the embedding layer gives each a rich numeric meaning that captures similarity and usage. Everything downstream operates on these vectors, so this humble table is where language first becomes math.
For product teams
The unglamorous doorway where words become numbers the model can reason over.
For engineers
A learned lookup table mapping token IDs to dense vectors, trained jointly with the model.
Related
- Embeddings — The vectors it produces.
- Tokenization — What feeds into it.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome