What does multimodal mean?
A modality is a type of data: text, image, audio, video. A multimodal model takes in or produces several of them directly — not by having another program describe the picture to it first, but by seeing it itself.
That difference is the whole point. The old approach chained separate systems together: an image recogniser described the picture, and the language model received the description. Information was lost at every step, and the language model only ever saw what the other program thought was important.
In a multimodal model there is no translation step. The image and the text go into the same internal representation, and the model works on both at once. That is why it can do things the chain could not: you can ask about a tiny detail in the picture, compare the difference between two images, or ask what is missing from one.
What it can actually do today
Images as input — this is the most mature. Photograph a paper invoice and get it back as a table. Paste in a screenshot of an error message and it tells you what is wrong. Hand over a hand-drawn sketch and it becomes HTML. Photograph a dashboard, a floor plan, a handwritten note.
Audio as input — reliable, but language-dependent. Transcription in English is excellent. In smaller languages it is good, but technical terms and proper names are worth reviewing afterwards. If several people talk over each other, separating the speakers is still the weak point.
Audio as output — natural, but not perfect. Read-aloud is no longer robotic. Emphasis can still slip in long, complex sentences, and foreign names are sometimes mispronounced.
Video — the least mature. Several models accept video as input, but typically by breaking it into frames. That means fast movement and continuous action can fall between the cracks.
What actually matters in daily work
Not that the model "sees". That typing disappears from the process.
Until now, if you wanted to work with a figure printed on paper, somebody had to type it in. That was the bottleneck: not the decision, not the analysis, but the data entry. Multimodality cuts out precisely that step.
Three places where this shows up in money straight away:
- Incoming documents — invoice, delivery note, minutes: from a photo to structured data.
- Fault reports — the customer photographs the problem instead of describing it.
- Notes taken in the field — a voice recording in the car, a finished report at the office.
Four things it is unreliable at
The limits are worth knowing, because it fails confidently.
Reading precisely from a dense table. In a photo of a crowded table the columns can bleed into each other, and the model will confidently print the wrong number. Check anything you are going to calculate with.
Handwriting. It copes well with printed text, less predictably with cursive — and it does not flag the errors.
Spatial relationships. "What is behind the leftmost box?" — it often gets questions like this wrong, even when everything is visible in the picture.
Counting things in an image. How many people in the photo, how many boxes on the shelf: a surprisingly weak point. If the number matters, do not have the model count it.
The same rule applies to all four as everywhere else: if you would not recognise a wrong answer, do not hand it the job. That is what Can AI Do This Task? is about.
How to ask about an image properly
Say what you are looking for. "What is in this picture?" gets you a general description. "What is the total and what is the payment deadline?" gets you what you need.
Ask for a format. "Give it back as a table with columns: item, quantity, net, VAT." That way the result can be passed on.
Give it a foothold. "This is a delivery note in Hungarian; the date format is YYYY.MM.DD." Now the model is not guessing at the format.
Ask it to flag uncertainty. "Where it is not clearly legible, write [illegible] rather than guessing." That one sentence catches more errors than anything else.
What to take away
Multimodality is not useful because it is impressive. It is useful because paper and speech become inputs, with no typing. Find the point in your processes where somebody types today because the data arrives in the wrong format — that is where the real gain sits.