Decoder. plain-English AI glossary

Model serving

● Core

Running a model as a live service that accepts requests and returns predictions, rather than training or batch processing.

Think of it like

Running a restaurant open for lunch instead of just cooking at home for family.

Example

Serving a model via an API that clients POST prompts to and get back completions.

How it actually works

Model serving requires orchestrating compute, managing the model's memory footprint, handling concurrent requests, and keeping latency low. You need to pick hardware (GPU, CPU, TPU), choose inference software (vLLM, TensorRT, etc.), and set up monitoring and autoscaling.

For product teams

Model serving is where inference goes live and starts costing money proportional to usage.

For engineers

Optimize for throughput and latency; batch where you can; manage memory carefully.

Endpoint — The URL where you send requests.

Read anything AI without the jargon

Look up any term in plain English, or save terms as you read with the free Chrome extension.

Open Decoder Add to Chrome