Plan a LoRA fine-tune for Mixtral that leaves the router undisturbed.

Short answer·Medium·4.0 · 0·~3 min·Asked atDatadog Haptik Lambda Labs

Attempt it

You must adapt Mixtral 8x7B to a customer support domain on limited GPUs. A previous full fine-tune attempt destabilized routing and tanked quality. Plan a LoRA based setup: where do you attach adapters, what do you freeze, and why does this protect the router?

Free · 2 AI evals / day

**LoRA** attaches small low-rank adapters to chosen weight matrices, learns only those adapters, and leaves the base model frozen. The decisions are which matrices to attach to and which to leave untouched. For Mixtral the answer is: attach to the dense attention projections (q, k, v, o). These weights are shared by every token regardless of routing, so adapter updates accumulate signal cleanly from your dataset and the domain shift gets absorbed there. **Freeze the router.** The router is a small linear layer whose job is to send each token to the right expert. Its weights encode a routing pattern that pretraining built up from trillions of tokens. A narrow domain dataset of ten thousand support tickets does not have the diversity to retrain it, so any update will drift the routing toward whichever expert happens to dominate your sample. The previous full fine-tune destabilized exactly because the router was learning. Lock the gate, and the experts keep their pretrained specialization. If your compute budget allows, you can also attach one small adapter per expert FFN. One adapter per expert preserves expert distinctness. A single shared adapter across experts would dilute specialization.

**The plan in three parts.** **Where to attach.** Put LoRA adapters on the dense attention projections (q_proj, k_proj, v_proj, o_proj). These are shared by every token, so every batch contributes to every adapter's gradient. The signal to noise on adapter learning is high. **What to freeze.** Freeze the **router weights** absolutely. Freeze the base weights everywhere (LoRA's default). Optionally, attach one small adapter per expert FFN. Critical: one per expert, not one shared across experts. A shared adapter would average updates across specialized experts and erase the specialization that took pretraining to build. **Why this protects the router.** With a customer-support dataset of modest size, the token distribution is narrow. Router gradients see this narrow slice and update toward whichever experts dominate it. After enough steps, the gate sends almost everything to one or two experts. That is exactly the failure your earlier full fine-tune hit. Frozen gates keep token-expert paths stable, and the adapters then learn the domain on top of consistent expert behavior. **Verification.** Sample a batch of support queries and a batch of pretraining-like prose. Log per-expert utilization before and after fine-tuning. Pretraining-like prose should produce the same utilization distribution as the base model. If it drifts, your routing was not actually frozen.

TL;DR

Freeze the gate, put LoRA on attention projections (and optionally one adapter per expert), so a narrow domain dataset cannot rewrite the routing that took pretraining to learn.

Memory aid

Easy to grasp

A Mixture of Experts model is like a restaurant with a host at the door and eight specialist chefs in the kitchen. The host has spent years learning which chef should cook which dish. If you take the restaurant and try to teach it a new cuisine using only a hundred sample meals, the host could easily get confused and start sending every order to one chef. The fix is to lock the host in place during retraining. The chefs can pick up a few new techniques, and you can tweak the recipes a little, but the door policy stays exactly as it was. That way the careful matching of dish to chef survives.

Key concepts

Mixture of Experts

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Fine-tuning an MoE is qualitatively different from fine-tuning a dense model because the routing gate sits between the model and every prediction. A tiny update to the gate changes which expert sees every token from that point forward, which in turn changes which gradients flow back into which expert weights. The system is feedback-coupled in a way no dense model is.

This deep dive lays out the LoRA plan for Mixtral, names the failure mode the user already hit, and walks through the verification steps that prove the routing stayed put.

What pretraining built into the router

Mixtral's router was trained for trillions of tokens spanning every text distribution the Mistral team could assemble: code, prose, math, dialogue, multiple languages. The optimizer balanced two pressures. The load-balancing auxiliary loss pushed the gate toward uniform expert utilization. The downstream cross-entropy loss pushed each expert toward useful specialization on its share of inputs.

The result is a gate that maps token features to a roughly uniform distribution across experts on broad inputs, while routing structurally similar tokens to the same expert. Code tokens cluster on certain experts. Mathematical notation clusters elsewhere. This specialization is what makes the sparse computation worth the architectural cost.

That structure is fragile under narrow fine-tuning. If you fine-tune on ten thousand customer-support tickets, the router sees support-flavored tokens and only support-flavored tokens. The pressure to balance is gone (auxiliary loss is often turned off during fine-tuning, and even if on, the dataset is too small for its signal to matter). The pressure to specialize collapses to whichever experts happen to handle support tokens best. The gate drifts toward those experts. Within a few thousand steps it can collapse to top-1 on a single expert.

Where LoRA should attach

Freezing the gate (and verifying it stayed frozen)

Auxiliary loss, capacity factor, and inference parity

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Mistral AI's official Mixtral fine-tuning guidance for PEFT recommends targeting attention projections and freezing gate weights
PEFT library in HuggingFace supports per-expert LoRA on Mixtral via target_modules filters that exclude router parameters

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would the plan change if you needed multilingual adaptation rather than domain adaptation?

Multilingual data may activate genuinely different experts. You may need expert adapters more than attention adapters, but the gate should still be frozen unless the language was severely underrepresented in pretraining.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Letting router weights receive gradient updates during fine-tuning, which rewrites learned token-expert assignments and destroys specialization.

60 second bullets to scan on the way to the call.

Why the router is treated differently from other model weights in MoE fine-tuning
What target modules a LoRA config should and should not include

Primary sources. Browse if you want the original framing.