Early Fusion
Combining raw data from multiple modalities before any processing - concatenating inputs and processing jointly.
Think of it like
Mixing all ingredients at the start of cooking, then heating together.
Example
Text tokens + image patches -> concatenate and feed to transformer. Audio waveform + visual frames -> joint processing.
How it actually works
Pro: joint optimization from the start, exploits cross-modal correlations early. Con: modalities have different dimensionality, noise, and temporal alignment issues; early concatenation may not work well. Requires careful preprocessing (time-syncing audio/video, resizing images). Good when modalities are naturally aligned (e.g., talking head: audio + video frames are synchronized).
For product teams
Works well when modalities are perfectly synchronized; simpler architecture.
For engineers
Concatenate features, process with shared backbone (CNN, transformer). Challenge: balancing modality sizes and rates.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome