Learn AI /Large Language Models /Vision-language model
Vision-language modelVLM
- Deeper
- Large Language Models
- Technical readers
- Programme & data teams
Also called: VLM, large multimodal model, LMM.
Back to the milestone you came from
In five seconds
What it is
A VLM links images and text so one model can take visual input, written input, or both, and produce text or other structured outputs about them.
In plain words
A VLM is an AI model that can use pictures and words together, so you can show it an image, ask a question in text, and get a text answer or extracted information.
An everyday example
Amina works at a service desk that receives phone photos of paper forms from offices in Nairobi, Bogotá and Amman. She looks at each photo, notices the layout, the boxes and the headings, and then matches it to one of several known templates. If the sender adds a note such as “find the invoice number”, she uses both the photo and the note to decide where to look. She then replies with the template name and the requested field value. If the photo is blurred or cropped, she may still give an answer that sounds confident but is wrong.
A person can use broad background knowledge, ask clarifying questions, and deliberately inspect uncertain areas. A VLM does not look at an image in the human sense. It processes learned numerical representations and generates outputs from patterns in training and current input. Also, many VLMs do more than template matching: they can answer open questions, summarise images, or combine several input types.
Remember thisA VLM can condition its output on images and text together. It may still misread, omit or confidently infer visual details, so outputs must be checked against the source.
This term is contested
Usage varies across research papers and products. Some authors use VLM narrowly for vision-plus-language systems, while others use broader labels such as LMM or multimodal LLM for overlapping capability sets.
Understand it
Understand it
Why it matters
For non-specialists, this term matters because many new AI tools no longer work on text alone. If a tool can inspect images, forms, charts or screenshots, its risks and uses change. That affects procurement, privacy review, testing, accessibility planning and human oversight. Knowing the difference between a text-only model, a VLM and a broader multimodal model helps you ask the right questions about inputs, failure modes and whether a claimed capability is actually needed.
How a vision-language model combines image and text
After viewing this, a reader should be able to explain how image content and text are converted into a shared context and then used to generate an answer.
01
1. Inputs
The system receives an image, text, or both.
Draw two input cards on the left. One shows a small form image thumbnail. The other shows a text bubble: 'Which template is this? Extract invoice number.' Use arrows pointing right.
02
2. Separate encoding
The image is converted into visual features. The text is split into tokens and converted into text features.
Place two boxes to the right of the inputs. Label the upper box 'Vision encoder' with a grid icon. Label the lower box 'Text tokens/encoder' with token chips. Keep the arrows separate.
03
3. Shared context
The model places visual and text representations into one processing context so words can be linked to image regions or patterns.
Merge both arrows into a larger central box labelled 'Shared multimodal context'. Inside, show small image patches and token chips mixed together, with thin linking lines between some words and image areas.
04
4. Generation or extraction
The model predicts output tokens one by one, or produces structured fields such as labels or values.
Add a box on the right labelled 'Decoder / output head'. Show a short token sequence appearing in order and, beside it, a small key-value card: 'template: invoice_v3; invoice_number: 84731'.
05
5. Human check
People should review outputs when errors would matter, especially if images are poor quality or the task affects rights, payments or operations.
Place a final review box with a person icon and a warning badge. Add a note below: 'Low-quality image or high-stakes use -> verify before acting'.
A left-to-right flow diagram with two inputs at the left: an image and a text prompt. They pass through separate encoding stages, merge into one shared context, then move to an output stage that generates text or structured fields. Read from left to right to see what goes in, how it is combined, and what comes out.
On hover or tap, highlight the path for either 'image only', 'text only', or 'image + text' to show that the same overall system can work with different input combinations.
Describe this without the diagram
The diagram is a five-step flow from left to right. Step 1 shows two possible inputs: a form image and a text question asking for the template and invoice number. Step 2 shows separate processing: the image goes to a vision encoder and the text goes to token and text encoding. Step 3 merges both into a shared multimodal context, meaning the model can relate words to visual patterns. Step 4 shows output generation, either as normal text or as structured fields such as a template name and invoice number. Step 5 adds human review, emphasising that outputs can be wrong and should be checked in high-stakes cases or when image quality is poor.
With reduced motion: Show the same five boxes in a static left-to-right layout with visible arrows and labels for image input, text input, separate encoding, shared context, output, and human review. Instead of animated highlighting, use three small legend tags above the arrows: image only, text only, and image plus text.
A vision-language model, or VLM, is a model built to connect images and language. In practice, people also say large multimodal model, or LMM, for larger systems that handle several kinds of input. The terms overlap, but they are not always identical: a VLM specifically connects vision and language, while an LMM may also include audio, video or other data types.
What goes in can be an image, a question in text, or both. In one common generative design, a vision component turns the image into machine-readable features. The text is turned into tokens, which are small pieces of text a model processes. The visual and text representations are then supplied to a shared processing context so the model can relate words to image regions or patterns.
Many generative VLMs then work similarly to a language model, but with visual information added to the context. They predict text tokens one after another. Depending on the design, outputs may instead include labels, coordinates or extracted fields. For example, a VLM may answer a question about a chart, describe a photo, or read values from a scanned form.
What comes out is not guaranteed truth. A VLM can miss small details, misread blurry text, or confidently describe something not present. A VLM uses computer vision but specifically links visual processing with language. This differs from a text-only large language model, which cannot directly process an image unless another component first converts it into usable input.
How “Matching a photographed form to the right template” maps onto the real thing
| In the story | In the AI system |
|---|---|
| The phone photo of the form | The visual input to the model |
| The sender's note asking for a field | The text prompt that guides the output |
| Amina noticing layout and headings together | The model linking visual patterns with language representations |
| The reply with a template name or field value | The generated text or structured output |
A worked example
During acceptance testing at Headquarters, a finance assistant tests an invoice-capture tool on three synthetic templates before comments on the vendor demonstration are due at close of business.
Starting from: Image: a slightly tilted photograph of a synthetic invoice. Prompt: "Identify template A, B or C, then return only the invoice number."
- The vision pathway converts patches or regions of the photograph into numerical features representing layout, printed characters and field positions.
- The prompt is tokenised into text embeddings, including the permitted template labels and the requested field.
- The model projects or aligns the visual and textual representations into a shared context, allowing the instruction to condition attention to likely headings and form fields.
- The decoder generates the response token by token, first producing the template label and then the invoice number.
Result: The tool returns "Template B, TEST-84I". The finance assistant checks the synthetic source and finds that the final character is the digit 1, not the letter I. The format is correct, but glare and similar character shapes produced a plausible extraction error.
In a United Nations workplace
At a regional hub, the reporting officer supporting procurement is reviewing closed solicitation files before the retention register goes to the chief of section at close of business. In an approved system, the officer tests whether a VLM can read scanned cover sheets, distinguish document types and propose file references for human confirmation.
The model can use layout, visible text and the officer’s written instruction together. This may help sort mixed scans or locate a requested field, including where ordinary text search fails because the page is an image.
Before you try thisProcurement files may contain bids, evaluation material, pricing, personal details and commercially confidential information. They must not be uploaded to any tool that is not approved for that material, because disclosure could breach access controls or compromise a procurement process. Model-generated labels must be checked against the source file and cannot override retention requirements.
Where you may hear it
- A vendor demonstration during a procurement meeting
- An ICT governance board reviewing a multimodal tool
- A records-management meeting on classifying scanned files
- An evaluation read-out comparing extraction against verified source documents
Avoid confusion
Avoid confusion
Often confused with
Multimodal model
A multimodal model can take in or produce more than one kind of data, such as text, images or audio, within one system.
Computer vision
Computer vision enables software to analyse images or video and produce useful outputs such as labels, locations, measurements or text.
Large language model
A very large text-processing model predicts likely next pieces of language from a prompt and uses that pattern-matching ability to generate or transform text.
Common misconceptions
Common misconception: A VLM is just an ordinary language model with image files attached.
Correction: Some systems do bolt image processing onto a language model, but the key point is that visual information must be converted into representations the language side can use. The model's behaviour changes because the image becomes part of the context, not because the file is merely attached.
Common misconception: If it can describe an image fluently, it must be accurately reading the whole image.
Correction: Fluent output is not proof of accurate visual grounding. A VLM may rely on likely patterns, miss small text, confuse similar layouts or describe objects that are not there, especially with blur, glare, cropping or unusual formats.
Common misconception: VLM and multimodal model mean exactly the same thing.
Correction: They overlap, but not perfectly. A VLM specifically links vision and language. A multimodal model is a broader category for systems that work across more than one data type, which may include text, images, audio or video. Large multimodal model usually refers to larger general-purpose versions of that broader category.
Quick check
Which statement best captures the relationship between VLM and LMM?
This is the safest summary. In practice, people use the terms loosely, but VLM points specifically to vision plus language. LMM usually refers to broader large multimodal systems, which may include vision-language capability and sometimes more.
Go deeper
Go deeper
The simple version
A VLM is an AI model that can use pictures and words together, so you can show it an image, ask a question in text, and get a text answer or extracted information.
The technical version
A VLM combines a vision pathway with a language model or language-like decoder. The image is transformed into numerical features, often from patches or regions, while the text is tokenised into text embeddings. These representations are aligned or projected into a shared space or shared context so the generation component can condition on both. In many implementations, the final output is still produced autoregressively through next-token prediction, even when the task looks like image understanding. 'Large multimodal model' is often used for broader systems that may support text plus images and sometimes audio or video; 'VLM' is narrower because it specifically names the vision-language pairing. The exact architecture varies, so the term refers more to capability class than to one single design.
Reviewed external resources
EncyclopaediaVision-language model (opens in a new tab)Wikipedia
This link gives a broad encyclopaedic overview of the term and suits readers who want a quick orientation before going deeper.
Intermediatelink checked 2026-07-31
CourseLLM Course (opens in a new tab)Hugging Face
This course link is useful for readers who want practical and technical background on modern model architectures and multimodal extensions.
Deeperlink checked 2026-07-31
VideoLarge language models, explained visually (opens in a new tab)3Blue1Brown
This video gives a visual explanation of how language-model mechanisms work and suits readers who want intuition before adding vision-specific detail.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Vision-language model (opens in a new tab)YouTube search
This search results page helps readers find multiple video explanations of vision-language models and suits those who prefer comparing teaching styles rather than relying on one source.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Vision-language model (opens in a new tab)Reddit search
This search results page helps readers see community questions, examples and debates about vision-language models, which suits readers who want informal discussion with caution about reliability.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
A VLM can condition its output on images and text together. It may still misread, omit or confidently infer visual details, so outputs must be checked against the source.