ControlNet
Adding spatial control to image generation—guide the diffusion process with a sketch, pose, or depth map.
Think of it like
Giving the AI artist an architectural blueprint before they paint.
Example
Sketch stick-figure poses; ControlNet generates realistic people in those poses. Provide edge map; generates images matching those edges.
How it actually works
Additional control branch learns to inject spatial conditioning (canny edges, pose keypoints, depth) into diffusion UNet. Preserves model quality while adding fine-grained spatial control. Separate ControlNets for different conditions; can stack multiple controls.
For product teams
Drastically improves usability—artists can sketch rough layouts, model fills in details.
For engineers
Conditioning encoder processes control signal (sketch, pose, depth); injected into UNet via cross-attention and additive conditioning at multiple scales.
Related
- Diffusion — Extension of.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome