Decoder. plain-English AI glossary

Serving

● Core

Running a trained model in production to answer real user queries.

Think of it like

Opening a restaurant after months of cooking in your home kitchen.

Example

You train a summarization model and deploy it on AWS behind an API so users can call it.

How it actually works

Serving is where the rubber meets the road. Training a model is one problem; serving it to millions of users on low latency with high reliability is another. You need caching, batching, load balancing, and fallbacks. A model that works in training can be slow, memory-hungry, or unreliable at scale. Most production failures are in serving, not modeling.

For product teams

Serving costs matter. A slow model is a bad model.

For engineers

Use optimized frameworks (vLLM, TensorRT). Profile latency and throughput. Implement monitoring and rollback.

Where models go to work.

Read anything AI without the jargon

Look up any term in plain English, or save terms as you read with the free Chrome extension.

Open Decoder Add to Chrome