Decoder. plain-English AI glossary

llama.cpp

▲ Rising

A lightweight C++ inference engine optimized for running GGUF models on consumer hardware (CPUs and weak GPUs).

Think of it like

The Honda Civic of LLM inference engines — small, reliable, and works on anything.

Example

Run a 7B-parameter model on a MacBook or Linux laptop using llama.cpp and get 10+ tokens per second.

How it actually works

Llama.cpp is pure C++ with SIMD optimizations (AVX2, NEON) for CPU inference. It's also got basic GPU support (CUDA, Metal, Vulkan) and can run on almost any device from phones to servers. The main trade-off is latency — it's slower than vLLM or TensorRT but doesn't need GPUs.

For product teams

Llama.cpp is the gateway to running open LLMs locally — privacy and cost matter.

For engineers

Llama.cpp is easy to deploy and has solid community momentum; it's your go-to for CPU inference.

GGUF — GGUF format that llama.cpp uses.

Read anything AI without the jargon

Look up any term in plain English, or save terms as you read with the free Chrome extension.

Open Decoder Add to Chrome