Flash Attention
A faster, memory-frugal way to compute exact attention by being clever about how the GPU moves data.
Think of it like
Cooking straight from the counter instead of walking to the pantry for each ingredient — same dish, far less trips.
Example
Swapping in flash attention lets a team train on longer sequences at the same speed, with no change to the results.
How it actually works
Standard attention writes a giant scores matrix to slow GPU memory and reads it back — the real bottleneck. Flash attention tiles the computation so intermediate values stay in fast on-chip memory, computing the same exact result without ever materializing the full matrix. The output is identical; the speed and memory savings are large, which is why it’s become the default.
For product teams
An under-the-hood efficiency win that made longer contexts and cheaper training practical.
For engineers
An IO-aware, tiled exact-attention kernel that avoids materializing the full attention matrix in HBM.
Related
- Attention — A faster computation of attention.
- Training — Speeds up training.
- Context length — Helps enable long context.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome