Local Inference
Running model inference on your own machine or private infrastructure, not on a cloud providers servers.
Think of it like
Storing documents in your filing cabinet instead of using a cloud service.
Example
Using ollama to run an open-source LLM on your laptop for RAG without paying per API call.
How it actually works
Includes both on-device inference and self-hosted servers. Requires ownership of compute hardware, model weights, and infrastructure. Offers full control and data privacy but demands operational expertise and hardware investment.
For product teams
Locks in infrastructure cost but eliminates per-token variable costs and prevents vendor lock-in.
For engineers
Inference loops run on self-managed compute—bare metal, containerized, or minimal orchestration—not via third-party APIs.
Related
- Inference — Complementary to.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome