Token streaming
Returning generated tokens one-by-one as they're produced instead of waiting for the entire response.
Think of it like
Getting subtitles in real-time during a movie instead of waiting until it's over.
Example
In a chat app, you see words appear as the model generates them rather than a 5-second pause then all text at once.
How it actually works
Token streaming is both a user-experience win (feels responsive) and an implementation challenge (need persistent connections). The server keeps the connection open and pushes tokens via SSE, WebSocket, or gRPC streaming. Clients need to handle backpressure if they can't consume tokens fast enough.
For product teams
Token streaming is the 'secret sauce' that makes LLM chats feel snappy.
For engineers
Use SSE or WebSockets; make sure your inference engine can stream tokens incrementally.
Related
- Streaming — Sending responses as they generate.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome