- 1Serve the model for multimodal Q&A and dialogue
- 2Visual instruction tuning on image grounded Q&A data, with the LLM typically unfrozen
- 3Collect and curate image caption pairs for feature alignment
- 4Freeze the pretrained vision encoder and the LLM
- 5Train only the connector/projection to align visual features to the LLM's embedding space
Gather caption pairs, freeze the encoder and LLM, train only the connector to align features, then unfreeze the LLM for visual instruction tuning, then serve — alignment must precede instruction tuning.
Imagine hiring a translator to sit between a photographer and a writer who speak different languages. First you train just the translator on simple picture and caption pairs, while the photographer and writer rest. Once the translator can pass meaning across, you let the writer practice answering real questions about the photos. The order matters. You cannot run a real question and answer session until the translator can actually carry the meaning. So the cheap translator training comes first, then the harder practice, and only then do you open the studio to customers.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Stitching a vision encoder onto a language model sounds like a wiring job: connect the two and train. The reason it is an interview question is that the wiring order is load-bearing, and getting it wrong produces a model that technically runs but quietly ignores the image.
The projection-style recipe, made famous by LLaVA, splits training into stages defined by what is frozen and what data flows in. Each stage has a narrow job, and the stages must run in a specific order because a later stage assumes the earlier one already succeeded. An ordering question tests whether you understand that dependency, not whether you can list steps.
This deep dive walks the five steps in sequence, explains what freezing buys at each point, shows why the cheap alignment stage has to come before the expensive instruction-tuning stage, and what concretely breaks if you flip them. The thread throughout is a dependency chain: the connector must speak the LLM's language before the LLM can be taught to reason in it.
The three moving parts and why freezing matters
A projection-style VLM has three pieces. The vision encoder, usually a CLIP-pretrained ViT, turns an image into a sequence of feature vectors. The LLM is a pretrained language model that cost enormous compute to train. Between them sits the connector, a small projection or MLP, with far fewer parameters than either neighbor — often a fraction of a percent of the whole system.
Freezing is the central lever. A frozen module still contributes features and gradients still flow through it to reach trainable parts upstream or downstream, but its own weights do not update. So freezing is not the same as removing a component; it means the part participates in the forward pass but is held fixed. Because the connector is tiny, training only it is cheap and stable — there is little to overfit and little optimizer state to carry per step.
The encoder and LLM are frozen early for a protective reason. Their pretrained weights encode hard-won general capability that took massive data and compute to acquire. Letting gradients from a not yet aligned visual stream rewrite them risks degrading that capability before the connector even works. So the recipe spends the first stage moving only the cheap part, and defers touching the expensive parts until the inputs they receive are trustworthy. Every later freezing decision in the recipe traces back to this same instinct: protect the costly models from noisy early gradients.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LLaVA — the canonical two-stage recipe: pretrain the projection on caption pairs with everything else frozen, then visual instruction tune with the LLM unfrozen.
- LLaVA-1.5 / NeXT — extend the same spine with better instruction data and a high-resolution stage, keeping align then instruction tune intact.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy keep the vision encoder frozen even during instruction tuning in many recipes?
Discuss preserving the encoder's pretrained features, limiting compute, and avoiding overfitting the encoder on a small instruction set, plus when a later encoder-unfreezing stage is added.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Flipping the order — instruction tuning before alignment. The LLM gets gibberish visual tokens early and learns to ignore the image instead of grounding answers in it.
60 second bullets to scan on the way to the call.
Why image-caption data comes first for feature alignment
Which components are frozen during the connector-only stage
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.