VLM
Neural network that understands both images and text—can caption photos, answer questions about images, or generate images from descriptions.
Think of it like
A person who can read, write, and understand pictures—richer than someone who only reads or only looks.
Example
GPT-4V takes an image and asks "what is in this photo?" and responds "a beach sunset with people swimming"; CLIP retrieves images matching text queries.
How it actually works
Vision encoder (ResNet, ViT) extracts image embeddings; text encoder converts words to embeddings; both sit in same space for alignment. Training requires image-text pairs (billions from web-scraped captions). More powerful than separate vision and text models but more compute-intensive.
For product teams
Enables richer user experiences: visual search, image Q&A, content understanding for moderation.
For engineers
Two-tower architecture: image tower outputs embeddings, text tower outputs embeddings; contrastive loss aligns them; downstream tasks use aligned space.
Related
- Vision Model — Combines.
- Language model — and.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome