In LoRA, where in the forward pass does `lora_dropout` apply its mask?

Attempt it

Quick reference

TL;DR

LoRA dropout masks the input activations to the A matrix, regularizing the trainable low-rank branch while leaving the frozen base weight untouched.

Memory aid

Easy to grasp

Think of LoRA as a small note-taker sitting next to a big locked encyclopedia. The encyclopedia is the frozen base model, and the note-taker is the tiny pair of matrices that learn the new task. Dropout is a daily exercise where the note-taker randomly covers up some of the words on the page before reading. Today certain words are hidden, tomorrow different ones. Because the note-taker never knows which words will be hidden, they have to learn the general shape of the message rather than rely on any single word. The encyclopedia itself is never touched, only the way the note-taker reads from it. That is why LoRA dropout sits at the start of the note-taker's reading path, not on the encyclopedia and not at the very end where the note-taker finishes writing.

Key concepts

LoRA adapters Dropout regularization Parameter-efficient fine-tuning Regularization

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: place dropout on the LoRA forward path, walk the input to A choice, contrast with output of B, list typical values, and link dropout, rank, and dataset size.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face PEFT exposes lora_dropout as a top-level field in LoraConfig and applies the mask before the A projection at every forward pass.
Axolotl and LLaMA-Factory recipes for Llama 4 SFT default lora_dropout to 0.05 across QLoRA and full LoRA configurations.
Unsloth tutorials for fine-tuning Mistral and Gemma 4 keep lora_dropout at 0.05 to 0.1 for instruction tuning on small curated datasets.
DeepSeek V4 distillation recipes raise lora_dropout to 0.1 when using rank 64 adapters to combat overfitting on student data.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does lora_dropout interact with the alpha over r scaling factor when you raise rank?

QWhy is dropout placed at the input of A rather than between A and B?

QHow would you decide between raising lora_dropout and lowering rank to combat overfitting on a small SFT set?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming dropout is applied to the base weight or to the output of the low-rank branch. It actually masks the input activations before they reach the A matrix in the trainable branch.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.