Multimodal input
Feeding a model more than just text — images, audio, or video alongside words.
Think of it like
Like talking to someone who can both read your note and look at the photo you’re pointing at.
Example
You upload a screenshot of an error and type “what’s wrong here?” — the model reads the image and the text together.
How it actually works
Multimodal models encode each input type into a shared representation space so the model can reason across them. It unlocks tasks pure text can’t touch — describing images, reading charts, transcribing speech — but each modality adds its own tokenization, cost, and failure modes, and image tokens can eat context fast.
For product teams
Opens whole product categories (photo Q&A, document understanding) but watch the token and latency cost.
For engineers
Inputs of mixed modalities encoded into a shared representation for cross-modal reasoning; images consume many tokens.
Related
- Representation — The shared space they’re mapped into.
- Token limit — The budget images strain.
- Embeddings — Vectors as the common currency.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome