Modality
A type of input or output - text, image, audio, video - that the model can process or produce.
Think of it like
Like your five senses: sight, hearing, touch, taste, smell. Each is a different modality, each carries unique information.
Example
GPT-4 Vision accepts both text and image modalities as input. A music generation model outputs audio. Multimodal models work across several at once.
How it actually works
Models are trained on specific modalities that match their training data and architecture. Fusing modalities - combining text + image, for instance - requires careful alignment so the model learns to reason across them. Each modality has different dimensionality and structure: text is discrete tokens, images are spatial, audio is temporal.
For product teams
Expanding to new modalities (video, audio) unlocks new use cases and markets, but raises development and training cost.
For engineers
Modality support requires specialized encoders/decoders and often separate embedding spaces that must be bridged.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome