Plan a LoRA fine-tune for Mixtral that leaves the router undisturbed.

Short answer·Medium·4.0 · 0·~3 min·Asked atMistral AI

Attempt it

You must adapt Mixtral 8x7B to a customer-support domain on limited GPUs. A previous full fine-tune attempt destabilized routing and tanked quality. Plan a LoRA-based setup: where do you attach adapters, what do you freeze, and why does this protect the router?

Free · 2 AI evals / day

**LoRA** attaches small low-rank adapters to chosen weight matrices, learns only those adapters, and leaves the base model frozen. The decisions are which matrices to attach to and which to leave untouched. For Mixtral the answer is: attach to the dense attention projections (q, k, v, o). These weights are shared by every token regardless of routing, so adapter updates accumulate signal cleanly from your dataset and the domain shift gets absorbed there. **Freeze the router.** The router is a small linear layer whose job is to send each token to the right expert. Its weights encode a routing pattern that pretraining built up from trillions of tokens. A narrow domain dataset of ten thousand support tickets does not have the diversity to retrain it, so any update will drift the routing toward whichever expert happens to dominate your sample. The previous full fine-tune destabilized exactly because the router was learning. Lock the gate, and the experts keep their pretrained specialization. If your compute budget allows, you can also attach one small adapter per expert FFN. One adapter per expert preserves expert distinctness. A single shared adapter across experts would dilute specialization.

**The plan in three parts.** **Where to attach.** Put LoRA adapters on the dense attention projections (q_proj, k_proj, v_proj, o_proj). These are shared by every token, so every batch contributes to every adapter's gradient. The signal to noise on adapter learning is high. **What to freeze.** Freeze the **router weights** absolutely. Freeze the base weights everywhere (LoRA's default). Optionally, attach one small adapter per expert FFN. Critical: one per expert, not one shared across experts. A shared adapter would average updates across specialized experts and erase the specialization that took pretraining to build. **Why this protects the router.** With a customer-support dataset of modest size, the token distribution is narrow. Router gradients see this narrow slice and update toward whichever experts dominate it. After enough steps, the gate sends almost everything to one or two experts. That is exactly the failure your earlier full fine-tune hit. Frozen gates keep token-expert paths stable, and the adapters then learn the domain on top of consistent expert behavior. **Verification.** Sample a batch of support queries and a batch of pretraining-like prose. Log per-expert utilization before and after fine-tuning. Pretraining-like prose should produce the same utilization distribution as the base model. If it drifts, your routing was not actually frozen.

Quick reference

TL;DR

Freeze the gate, put LoRA on attention projections (and optionally one adapter per expert), so a narrow domain dataset cannot rewrite the routing that took pretraining to learn.

Memory aid

Easy to grasp

A Mixture of Experts model is like a restaurant with a host at the door and eight specialist chefs in the kitchen. The host has spent years learning which chef should cook which dish. If you take the restaurant and try to teach it a new cuisine using only a hundred sample meals, the host could easily get confused and start sending every order to one chef. The fix is to lock the host in place during retraining. The chefs can pick up a few new techniques, and you can tweak the recipes a little, but the door policy stays exactly as it was. That way the careful matching of dish to chef survives.

Key concepts

Mixture of Experts

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

7 minutes: explain LoRA basics, anchor on the router as the load-bearing component, lay out adapter placement, freezing rules, and per-expert vs. shared adapter math, then close with verification.

Real-world examples

Real products, models, and research that use this idea.

Mistral AI's official Mixtral fine-tuning guidance for PEFT recommends targeting attention projections and freezing gate weights
PEFT library in HuggingFace supports per-expert LoRA on Mixtral via target_modules filters that exclude router parameters
DBRX fine-tuning workflows from Databricks Mosaic ship templates that lock router weights by default
DeepSeek-V3 community fine-tunes apply LoRA only to attention and shared experts, never to the routing gate

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would the plan change if you needed multilingual adaptation rather than domain adaptation?

QWhat goes wrong if you use a single shared FFN adapter across all experts?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Letting router weights receive gradient updates during fine-tuning, which rewrites learned token-expert assignments and destroys specialization.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.