What does DoRA decompose, and why does it beat LoRA at the same rank?

**DoRA** stands for Weight-Decomposed Low-Rank Adaptation. It is a small but clever upgrade to LoRA, the standard way to fine-tune a big model cheaply. Start with a single column of a weight matrix. Think of it as an arrow in space. Any arrow has two properties: a length, meaning how big it is, and a direction, meaning which way it points. DoRA pulls these two apart and handles each one differently. The length, called the magnitude, is just one number per column. DoRA learns it directly because it is cheap to store and easy to adjust. The direction is the hard, high-dimensional part, so DoRA hands that job to LoRA's usual low-rank update. LoRA touches the direction only. Vanilla LoRA, by contrast, has one small low-rank update that must capture BOTH length and direction at once. When the rank is tiny, that update is starved for room, and it wastes some of that room recording length. The benefit: DoRA learns much more like full fine-tuning, where every weight is free to move. The gain is biggest at very low rank, and it costs almost nothing extra, just one scalar per column.

**DoRA** decomposes each weight column into two independent parts. The first is a magnitude, one scalar per output column. The second is a direction, the column normalised to unit length. The decomposition is: $$W = m \cdot \frac{V}{\lVert V \rVert}$$ DoRA learns the magnitudes directly, since each is just one number. It applies LoRA's low-rank update to the direction component V alone. So magnitude gets its own dedicated knob, and the scarce low-rank capacity is reserved entirely for direction. The motivation comes from analysing full fine-tuning. Full fine-tuning moves magnitude and direction with a distinctive, partly independent pattern. Vanilla LoRA folds both into one low-rank term and cannot reproduce that pattern faithfully. By splitting them, DoRA tracks full fine-tuning far more closely. The payoff concentrates at low rank. At r equal to 4 or 8, plain LoRA struggles because its tiny update must encode length and direction together. DoRA frees that update for direction only. At higher rank the gap shrinks, since LoRA then has enough capacity to encode magnitude implicitly. The extra cost is trivial, one scalar per column plus a forward-pass normalisation.