Edge deployment
Running the model on user devices or regional servers instead of a central data center — faster but limited by device compute.
Think of it like
Running a factory branch in each city instead of shipping everything from one central factory.
Example
Running a small LLM on a user's phone using llama.cpp instead of sending requests to a cloud API.
How it actually works
Edge deployment has latency and privacy benefits (no network round-trip, data stays local) but size and accuracy constraints (device has limited memory and compute). Good for on-device AI; bad for large models. Requires careful model optimization (quantization, pruning).
For product teams
Edge deployment improves privacy and latency, but limits model size and inference quality.
For engineers
For edge, prioritize model size and speed; 7B quantized runs on phones, 70B doesn't.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome