Cross-Attention Fusion
Using attention to let one modality query another - "which parts of the image are relevant to this text?" - enabling fine-grained fusion.
Think of it like
You read a question, then scan the image to find the most relevant parts, then reason over those.
Example
VQA: text query "Where is the dog?" attends to image regions; high attention to dog area, low to background.
How it actually works
Query modality attends to key/value modality: Attention(Q_text, K_image, V_image). Enables asymmetric interaction (text queries image, or vice versa). Multi-head attention captures different query-image relationships. Can stack (iterative cross-attention) for refinement. Widely used in CLIP, BLIP, ViLBERT. More flexible than early/late fusion, captures fine-grained alignment.
For product teams
More accurate multimodal understanding; enables interpretability (which image regions matter).
For engineers
Implement with multi-head cross-attention. Typical: text queries image (or vice versa). Bidirectional: both query each other.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome