Streaming
Sending tokens back to the client as soon as they're generated instead of waiting for the full response.
Think of it like
Watching a video as it downloads instead of waiting for the whole file.
Example
A chat interface shows words appearing one by one as the model generates them, instead of a long pause then all text at once.
How it actually works
Streaming trades off latency perception for architectural complexity. The client gets feedback immediately (feels fast), but you need to maintain the connection and send partial data. Server-Sent Events (SSE) or WebSockets are typical. Generators/iterators on the server simplify implementation.
For product teams
Streaming makes LLMs feel snappy — users see progress instead of staring at a spinner.
For engineers
Streaming requires careful connection management and error handling; SSE is simpler than WebSockets.
Related
- Server-sent events — Protocol for streaming responses.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome