On-Device Inference
Running the model directly on your phone, laptop, or edge device instead of sending requests to a server.
Think of it like
Like keeping a calculator in your pocket instead of calling a math hotline every time you need an answer.
Example
Running Llama 2 on an iPhone to power offline voice commands without internet.
How it actually works
Trades server latency and privacy concerns for local latency and device compute constraints. The model fits in available VRAM, usually through quantization and optimization. Requires careful memory management and cannot leverage cloud scaling.
For product teams
Enables privacy-preserving features and offline functionality, but constrains model capability to what fits locally.
For engineers
Model inference executes on the users device CPU/GPU using optimized frameworks like Core ML, ONNX, or llama.cpp.
Related
- Inference — Reduces latency and privacy risk vs.
- Quantization — Often paired with.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome