LoRA dropout masks the input activations to the A matrix, regularizing the trainable low-rank branch while leaving the frozen base weight untouched.
Think of LoRA as a small note-taker sitting next to a big locked encyclopedia. The encyclopedia is the frozen base model, and the note-taker is the tiny pair of matrices that learn the new task. Dropout is a daily exercise where the note-taker randomly covers up some of the words on the page before reading. Today certain words are hidden, tomorrow different ones. Because the note-taker never knows which words will be hidden, they have to learn the general shape of the message rather than rely on any single word. The encyclopedia itself is never touched, only the way the note-taker reads from it. That is why LoRA dropout sits at the start of the note-taker's reading path, not on the encyclopedia and not at the very end where the note-taker finishes writing.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LoRA is a small surgical addition to a frozen transformer. A pair of low-rank matrices learns a correction that is added to a frozen weight at each layer. The whole structure is designed so that the bulk of the model never moves and only the tiny adapter trains. That design constraint is what determines where dropout can live and where it absolutely cannot.
The question hides a simple test of mechanism. Three of the four options place dropout somewhere structurally impossible or strictly worse. Only one places it where the math and the convention agree. Knowing which place that is tells an interviewer whether you have actually read a LoRA implementation or only the slogan.
This deep dive walks the forward pass, places the dropout precisely, explains why each alternative is wrong, and connects the placement choice to the way standard dropout is used elsewhere in transformer training. By the end, the answer should feel like a structural consequence rather than a memorised fact.
The LoRA forward pass and where dropout fits
A LoRA wrapped linear layer keeps the original weight W_0 frozen and adds a learnable correction expressed as a product B A. The forward computation is:
Here x is the input activation, A is a small matrix mapping from input dimension d down to rank r, B is a matrix mapping from rank r back up to output dimension k, and the scalar alpha over r controls the magnitude of the adapter contribution. The frozen path on the left never trains.
Why the dropout has to live on the right
Dropout regularizes trainable parameters. Applying it to anything that touches W_0 would change the gradient that flows back to weights that are not trained, which is meaningless: there is no parameter on that path to receive the regularization signal. So whatever dropout LoRA introduces must live on the right hand side, the trainable branch.
The right hand side has exactly three positions where a mask could sit. Before A, between A and B, or after B. The convention picks the first: before A. The masked vector is then fed to A, and the rest of the branch operates as normal:
The placement matches how dropout is used in vanilla transformer training, where each linear layer receives a dropped version of its input rather than producing a dropped output.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face PEFT exposes lora_dropout as a top-level field in LoraConfig and applies the mask before the A projection at every forward pass.
- Axolotl and LLaMA-Factory recipes for Llama 4 SFT default lora_dropout to 0.05 across QLoRA and full LoRA configurations.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does lora_dropout interact with the alpha over r scaling factor when you raise rank?
Note that the scaling factor already shrinks the adapter contribution as r grows with alpha fixed, and discuss why high rank often benefits from a slightly higher lora_dropout to control the now larger adapter capacity.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming dropout is applied to the base weight or to the output of the low-rank branch. It actually masks the input activations before they reach the A matrix in the trainable branch.
60 second bullets to scan on the way to the call.
Where in the LoRA forward pass dropout is applied
Why the frozen base weight is never touched by lora_dropout
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.