Inference cost
The cost of running predictions — usually per token or per request.
Think of it like
The cost of electricity to run your ChatBot, not the cost to train it.
Example
At $0.01 per 1K tokens, a 1M-token inference job costs $10.
How it actually works
Inference cost = (model weights × cost per token) + (infrastructure overhead). Inference is cheaper than training but happens constantly. As usage grows, inference cost dominates. Optimizations here have huge leverage.
For product teams
Inference cost determines unit economics — it's the main variable cost per customer.
For engineers
Optimize inference cost first; it scales linearly with usage.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome