Visual Grounding
Locating which region of an image a text phrase refers to - bridging language and vision.
Think of it like
Someone says "the red car on the left," and you point to it in the picture.
Example
"The dog wearing a hat" -> system highlights the bounding box or segment of the dog wearing a hat in the image.
How it actually works
Related to: VQA (answer question), referring expression (find object from description). Requires fine-grained alignment of text and image regions. Methods: detection-based (find all objects, match to phrases), alignment-based (compute attention of phrases to image regions). Challenges: compositional phrases ("red dog" is red + dog), ambiguity (multiple matching regions), rare objects.
For product teams
Accessible search ("show me the person on the right"), interactive editing, multimodal retrieval.
For engineers
Input: image + text phrase. Output: bounding box or segmentation mask. Datasets: Flickr30K, ReferIt, RefCOCO. Metric: Intersection over Union (IoU).
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome