Decoder. plain-English AI glossary

Tokens per second

● Core

How many output tokens your inference server produces per second.

Think of it like

Words per minute for a typist — measures output speed.

Example

Your model generates 50 tokens per second. A 100-token response takes 2 seconds.

How it actually works

TPS is limited by memory bandwidth (for large models) and compute. Bigger models have higher per-token latency. You increase TPS through batching (process multiple requests together) and parallelization. The tradeoff: batching increases latency per request. You want to batch enough to get good TPS without making individual requests too slow.

For product teams

Throughput scales your revenue. More tokens per dollar means cheaper service.

For engineers

Measure on realistic batch sizes. Profile memory bandwidth. Use pipelining and tensor parallelism.

Throughput metric.

Read anything AI without the jargon

Look up any term in plain English, or save terms as you read with the free Chrome extension.

Open Decoder Add to Chrome