Prompt-tuning vs LoRA: pick the architectural reason LoRA tends to win

Both **prefix tuning** and LoRA are cheap ways to adapt a frozen base model without retraining all its weights. They differ in where they inject the change. Prefix and prompt tuning learn a small set of continuous vectors that act like an invisible prompt. These vectors sit at the input or get prepended to the attention keys and values. The actual weight matrices never move. Every adaptation has to flow into the model through attention, which is a narrow channel. LoRA takes a different route. It adds a small low-rank update directly to chosen weight matrices, usually the attention and MLP projections. So LoRA edits what the model computes at every layer it touches, not just what it reads at the start. That is why LoRA usually wins on quality. It has more leverage per trainable parameter. As a bonus, you can fold the LoRA update back into the base weights after training, so it adds no extra latency at inference. Soft prompts always cost a few extra tokens of compute.

Both methods are parameter-efficient fine-tuning. The difference is the part of the network each is allowed to edit. Prefix and prompt tuning learn continuous vectors. These get prepended to the input or to the attention keys and values. Every weight matrix stays frozen. The model is steered only through what it reads, so the adaptation has to squeeze through attention. That input-space bottleneck caps how much these methods can reshape the computation. **LoRA** edits the weights directly. The low-rank delta is: $$W = W_0 + \frac{\alpha}{r} B A$$ Its change lands inside the attention and MLP transforms at every adapted layer. So LoRA gets far more leverage per trainable parameter. That is the main reason it tends to win on quality. A secondary perk seals the choice. You can merge the LoRA delta back into the base weights after training. Inference then runs at zero extra cost. Soft prompts always consume a few key-value slots. So LoRA usually wins on both quality and serving cost, which is why it dominates open-weight adaptation today.