Text-to-Video
Creating a video from a text description—"a person dancing in the rain" produces a short video clip.
Think of it like
Writing a movie scene description that the AI director turns into footage.
Example
Sora takes "a white cat playing with yarn balls" and generates a 60-second video with realistic motion and lighting.
How it actually works
Extends image diffusion to temporal dimension. Model must learn frame-to-frame consistency, realistic motion, and temporal coherence. Latent diffusion (compress frames to latent space) is efficient. Training requires paired video-text data (billions of hours). Inference is slow (1 minute of video takes minutes to generate).
For product teams
Democratizes video creation; enables rapid prototyping and visualization.
For engineers
VAE compresses video frames; temporal UNet diffuses in latent space; CLIP guides; VAE decodes frames back to pixels.
Related
- Extension of image generation.
- Inverse of video understanding.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome