KL divergence
Also called Kullback–Leibler Divergence
A measure of how far one probability distribution has drifted from another — used as a leash to keep a fine-tuned model near its original.
Think of it like
A tether on a climber: it lets them move around but yanks back if they stray too far from base camp.
Example
In RLHF, a KL penalty rises as the tuned model's outputs diverge from the base model, discouraging it from going off the rails to chase reward.
How it actually works
KL divergence quantifies the difference between two distributions and is not symmetric — the direction matters. In RL fine-tuning it appears as a penalty term measuring how far the current policy has moved from the reference model; this keeps the model from exploiting the reward at the cost of coherence or safety. It also shows up broadly in training objectives like variational methods.
For product teams
The safety tether that stops reward-chasing from turning a good model into a weird one.
For engineers
A non-symmetric distance between distributions; used as a regularizing penalty against a reference policy in RLHF.
Related
- PPO — Uses it as a constraint each step.
- Cross-entropy — A related information-theoretic loss.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome