Cross-Modal
Connections between different modalities—image-text alignment, audio-visual sync, text-video coherence.
Think of it like
Knowing that a dog bark sound and a dog photo are related even though one is audio and one is visual.
Example
CLIP aligns image and text embeddings so "dog" text is close to dog photos; video model aligns audio speech with facial movements.
How it actually works
Learning alignment requires paired multimodal data. Contrastive learning pulls matching pairs together, pushes mismatches apart. Transformers with cross-attention learn fine-grained alignment (specific words to image regions). Not trivial because modalities have different information content and temporal dynamics.
For product teams
Enables richer search and understanding—find videos by audio description, caption images accurately.
For engineers
Cross-attention between modality encoders; contrastive loss during training; can use fusion early (concatenate) or late (combine embeddings).
Related
- Core concept in VLMs.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome