What does visual instruction tuning add on top of a pretrained vision encoder and LLM?

Short answer·Medium·4.0 · 0·~3 min·Asked atContextual Ai IBM Workday

Attempt it

You already have a strong vision encoder and a strong text LLM, and a connector that aligns them. Explain what visual instruction tuning does on top of that, what data it uses, and what behavior it actually unlocks.

Free · 2 AI evals / day

Start with what you already have. A strong vision encoder turns images into features, a strong LLM handles text, and a **connector** has been aligned so the LLM can read those visual features. After that alignment, trained on image-caption pairs, the model can describe an image — but mostly it just produces generic captions. The gap is that a captioner is not an assistant. If you ask it a specific question, or try to have a back and forth about the image, it tends to fall back on describing the whole scene instead of answering what you asked. **Visual instruction tuning** closes that gap. You train on image-grounded instruction and response pairs — questions about the image, multi-turn conversations, reasoning prompts — so the model learns to follow instructions and tie its answer to the specific image in front of it. This stage trains the connector and usually unfreezes the LLM too, so the LLM actually learns to use the visual tokens while it follows an instruction. The classic example is the LLaVA recipe, which uses generated instruction data over images to turn an aligned pair into a usable visual assistant.

Connector alignment and visual instruction tuning are two different stages, and the question is really about the second one. Alignment, trained on image-caption pairs, teaches the connector to project visual features into the LLM's token space so the model can talk about an image at all. But its behavior is captioning: hand it an image and it describes the scene generically, even when you wanted a specific answer. It matches images to text; it does not yet follow instructions. Visual instruction tuning trains on image-grounded instruction/response data — visual questions, multi-turn dialogue about the image, reasoning and explanation prompts. The objective shifts from describing the scene to doing what the user asked about this image. The model learns to follow multimodal instructions. Critically, it grounds its response in the specific image rather than emitting a one size fits all caption. Mechanically, this stage trains the connector and usually unfreezes the LLM, so the LLM updates to actually consume the visual tokens during instruction following rather than treating them as background. The canonical recipe is LLaVA: use a strong text model to synthesize instruction-style data over images (questions, conversations, detailed reasoning), then fine-tune on it. That step is what converts an aligned encoder-LLM pair into a genuinely usable visual assistant.

TL;DR

Visual instruction tuning trains on image-grounded questions and dialogue so the model follows instructions about an image instead of just captioning — turning an aligned encoder-LLM pair into an assistant.

Memory aid

Easy to grasp

Imagine you taught a student to label every photo with a caption — 'a dog on a beach.' Useful, but if you ask 'is the dog wet?' they just repeat the caption, because labeling is all they practiced. Visual instruction tuning is the next round of practice. You show the student photos paired with real questions, conversations, and reasoning tasks, and have them answer those. After enough practice, they stop reciting captions and start actually answering what you asked, looking at the specific picture in front of them. That practice is what turns a photo-labeler into a helpful assistant you can talk to about an image.

Key concepts

Multimodal AI Fine-Tuning Attention

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

It is tempting to think that once you have a great vision encoder, a great LLM, and a connector linking them, you have a visual assistant. You do not. You have a model that can talk about images in the blandest possible way — it captions. The leap from captioning to answering is the entire job of visual instruction tuning, and it mirrors the leap a base LLM makes when it becomes a chat model.

The question matters because the gap is invisible until you try to use the model. Alignment metrics look fine; the model clearly 'sees' the image. Then a user asks 'which of these two cables is HDMI?' and it replies with a paragraph describing the whole desk. That is the failure instruction tuning fixes.

This deep dive separates the two training stages, explains what kind of data teaches instruction following, walks through why the LLM has to be unfrozen, and ends on the LLaVA recipe — including the clever trick of generating instruction data from a text-only model that never saw the pixels.

Two stages, and why people collapse them

Building a VLM assistant is usually a two-stage fine-tune on top of a pretrained encoder and LLM, and the stages do different jobs.

Stage one, alignment, trains the connector — typically with the encoder and LLM frozen — on image-caption pairs. Its only goal is representational: get visual features into the LLM's token space so the LLM can reference them. After this stage the model can produce a caption for an image, which feels impressive and is why people stop here in their mental model.

But a captioner is not an assistant. Alignment optimized for matching images to descriptions, so the model's reflex is to describe. Ask a pointed question and it narrates the scene; start a conversation and it restarts from a caption each turn. The behavior you want — follow the instruction, answer this question, continue this dialogue — was never in the training signal.

That is the trap the question targets. Saying 'the connector is aligned, so the model follows visual instructions' skips the stage that actually teaches following. Alignment gives access to the image; it does not give instruction-following behavior over it.

The data that turns describing into answering

Why the LLM gets unfrozen

The LLaVA recipe and the data-generation trick

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

# Stage 1 — alignment: freeze encoder + LLM, train only the connector
vision_encoder.requires_grad_(False)
llm.requires_grad_(False)
connector.requires_grad_(True)

for image, caption in image_caption_pairs:        # captioning signal
    feats = vision_encoder(image)                 # patch features
    visual_tokens = connector(feats)              # project into LLM space
    loss = llm.lm_loss(visual_tokens, caption)    # learn to describe
    loss.backward(); optimizer.step()

# Stage 2 — visual instruction tuning: unfreeze the LLM too
llm.requires_grad_(True)                          # connector stays trainable

for image, instruction, response in instruction_data:  # QA / dialogue / reasoning
    visual_tokens = connector(vision_encoder(image))
    loss = llm.lm_loss(visual_tokens + instruction, response)  # learn to follow + ground
    loss.backward(); optimizer.step()

Real products, models, and research that use this idea.

LLaVA — the canonical recipe: synthesize instruction data over images with a strong text model, then fine-tune the connector and LLM into a visual assistant.
InstructBLIP — instruction-tunes a BLIP-2 backbone on a wide mix of vision-language tasks framed as instructions.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is synthetic instruction data, generated by a text-only model from image annotations, good enough to teach visual grounding?

Discuss feeding the text model rich annotations (captions, boxes) so it writes grounded questions, and why behavioral instruction-following transfers even though the generator never saw pixels.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing connector alignment with instruction tuning. Alignment teaches the model to match images and captions; instruction tuning teaches it to follow questions and dialogue grounded in the image.

60 second bullets to scan on the way to the call.

The difference between connector alignment and instruction tuning
Why an aligned model defaults to generic captioning

Primary sources. Browse if you want the original framing.