In LoRA, where in the forward pass does `lora_dropout` apply its mask?

**LoRA** keeps the base weight frozen and trains a small pair of matrices, A and B, whose product is added to the frozen weight at the forward pass. The full forward path is base times input plus the LoRA branch times input. The LoRA branch first projects the input down to a low dimensional space using A, then back up to the output dimension using B. The whole point is that A and B together are tiny compared to the base. Dropout is a regularizer that randomly zeros some entries of an activation vector during training. The natural place to apply it inside LoRA is on the input to the A matrix. Each step sees a randomly thinned view of the input, so both A and B have to learn to spread the signal rather than memorise specific positions. The frozen base weight is never touched by dropout. Dropping its inputs would mean retraining around it, which contradicts the choice to freeze it. Dropping the output of B would simply scale down the whole LoRA contribution rather than regularize how it learns.

**LoRA dropout** masks the input activations to the A matrix in the low-rank branch. The base weight is never touched. The forward computation for a LoRA wrapped linear is: $$h = W_0 x + \tfrac{\alpha}{r}\, B \, A \, x_{\text{dropped}}$$ where `x_dropped` is x with dropout applied. The dropout fraction is the `lora_dropout` config value, typically 0.05 to 0.1 for SFT. Each training step thins x differently, so A cannot memorise that a particular input dimension always carries useful signal, and B cannot rely on a particular column of the A times x intermediate. Dropping after B would scale down the final adapter contribution, which is closer to a weighting trick than regularization. Dropping the base input would defeat the whole purpose of freezing the base. Dropping gradients is not what dropout means; that would be a different intervention entirely. The practical effect is mild for low rank adapters, where the parameter budget is already small. It becomes meaningful when rank climbs into the dozens and the adapter starts to have enough capacity to overfit the SFT set.