Referring Expression
Understanding language that refers to specific objects in an image - "the woman in the red hat on the left."
Think of it like
In a crowded photo: "the person in the back row, third from the left."
Example
"The tallest building" -> output: bounding box of the tallest building in the image.
How it actually works
Comprehension task: given phrase, find object. Generation task: describe an object so someone can identify it. Comprehension requires grounding (localization). Generation requires visual understanding + language fluency. Compositional phrases ("red car" = red + car) and relational ("to the left of the dog") are challenging. Transformer-based models with cross-attention excel.
For product teams
Accessible systems, interactive retrieval, visual question answering.
For engineers
Comprehension: image + text -> bounding box/mask. Generation: image + bounding box -> description. Datasets: Flickr30K, ReferCOCO. Metric: accuracy, IoU.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome