inference provider
A company or service that runs LLM inference at scale and sells API access by the token.
Think of it like
A gas station for language models—you pay per gallon, they handle the pumps.
Example
Anthropic (Claude API), OpenAI (ChatGPT API), Together, Mistral, Groq. You send text, they compute the response, bill you.
How it actually works
Inference providers differ by model choice (which models do they offer), latency, pricing per token, rate limits, and feature support (streaming, function-calling, long context). Some (Anthropic, OpenAI) are vertically integrated; others (Together, Replicate) are aggregators or infrastructure plays.
For product teams
Shifts capex (buying GPUs) to opex (API spend); enables rapid experimentation.
For engineers
Abstracts away hardware; you focus on prompt and usage patterns.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome