AdaLoRA: how does adaptive rank allocation across layers work?

**LoRA** gives every adapted layer the same rank, say r=16. AdaLoRA notices this is wasteful, because different layers need different amounts of adaptation, and lets the layers compete for a shared budget. The trick is how it represents the update. Plain LoRA writes the weight change as a product of two matrices, B times A. AdaLoRA instead writes it in a form borrowed from singular value decomposition: two factor matrices with a small diagonal matrix in the middle. That diagonal holds a list of numbers, one per rank direction. Those diagonal numbers act like importance scores. A number near zero means its direction barely affects the model's output, so it can be dropped with little loss. A large number means that direction is doing real work and should be kept. During training, AdaLoRA periodically looks across the whole model and prunes the smallest diagonal entries. Pruning in one layer frees capacity that flows to layers whose entries survive. The total budget stays fixed, so the model ends up spending more rank where it helps and less where it does not. The cost is a more involved training loop and an extra penalty that keeps the factor matrices well-behaved. In exchange you get a smarter spend of the same parameter budget.

**AdaLoRA** keeps the LoRA idea of a low-rank weight update but reparameterises it and makes the rank adaptive per layer. It replaces the BA product with an SVD-style triple. The update is written as a left factor, a diagonal middle matrix, and a right factor. The diagonal entries play the role of singular values. Each entry gauges how much its direction contributes, so importance becomes readable per direction instead of tangled inside a matrix product. Training then prunes the low-importance diagonal entries on a schedule, judged by a sensitivity score rather than raw magnitude alone. The total number of surviving entries across the model is the rank budget. Pruning a useless direction in one layer hands that slack to a layer whose directions still matter. The allocation drifts toward actual per-layer need. Uniform rank is wasteful because layers differ in sensitivity. Early layers carry generic features and adapt little. Later layers and attention output projections sit closer to the task and want more rank. A flat r over-provisions the calm layers and starves the busy ones. AdaLoRA fixes that mismatch. The price is a heavier training loop and an orthogonality penalty on the two factors to keep the decomposition stable.