Speculative Architecture
Also called Speculative Decoding
A speed trick where a tiny fast model guesses several next words and a big model just checks them in one pass.
Think of it like
Like a junior drafts a few sentences and the senior editor approves or fixes them in a single quick read — faster than the senior writing every word.
Example
A serving stack might use a small draft model to propose 4 tokens, then have the big model verify all 4 at once.
How it actually works
The big model normally makes tokens one slow step at a time. Here a cheap draft model proposes a short run of tokens, and the big model verifies them in parallel, accepting the ones it agrees with and correcting at the first disagreement. Output is identical to the big model alone, just produced faster — pure latency savings with no quality loss.
For product teams
A way to make responses noticeably faster and cheaper with no drop in answer quality.
For engineers
Parallel verification of a small draft model’s proposed tokens by the target model, preserving its distribution.
Related
- Draft Model — The small model doing the guessing.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome