Cross-Attention
Attention from one sequence to a different one — letting an output peek at a separate input.
Think of it like
A translator reading the original sentence while writing each word of the translation.
Example
In a translation model, each output word cross-attends to the source sentence to stay faithful to it.
How it actually works
Cross-attention draws its queries from one sequence but its keys and values from another, so the first sequence conditions on the second. It’s how a decoder consults an encoder’s representation of the input, or how a model grounds generation in retrieved documents or an image. It’s the same math as self-attention, just pointed at a different source.
For product teams
The mechanism that lets a model’s output stay anchored to a separate input like a document or image.
For engineers
Attention where queries come from one sequence and keys/values from another, conditioning one on the other.
Related
- Self-Attention — Contrast with self-attention.
- Encoder-Decoder — Bridges an encoder and decoder.
- Attention — A form of attention.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome