Decoder. plain-English AI glossary

Prefix tuning

▲ Rising

A parameter-efficient method where you learn a short "prefix" of embeddings prepended to every layer, tuning the model for a task.

Think of it like

Whispering a hint in someone's ear before they answer every question.

Example

Train 100 vectors (the prefix) per layer to steer GPT-2 toward a task like summarization, using only 0.1% of the model's parameters.

How it actually works

Unlike full fine-tuning, prefix tuning keeps the model's parameters frozen and only optimizes a learned prefix. This is parameter-efficient, making it attractive for systems with many tasks. Downside: the prefix needs to be specific to the base model; changing the model requires retuning.

For product teams

Multi-task systems with shared base models and task-specific prefixes.

For engineers

Implement with learnable prefix tensors. Use gradient descent to optimize. Monitor for overfitting on small tasks.

Soft prompts — Related to soft prompts and prompt tuning.

Read anything AI without the jargon

Look up any term in plain English, or save terms as you read with the free Chrome extension.

Open Decoder Add to Chrome