Lora

Walk through what happens during backprop for a LoRA wrapped linear layer

Walk through the triage order when a customer support LoRA tanks MMLU by 4 points post-deploy

If LoRA freezes the base weights, does that mean activation checkpointing on the base is unnecessary?

What subtle problem do you hit when merging a LoRA back into an NF4 quantized base, and why?

What does Adam's two fp32 buffers per trainable parameter imply for LoRA vs full FT on a 7B?

Why might a LoRA that passes every eval check then garble one in five prod requests?

Estimate the trainable param count for a Llama-3 8B LoRA at r=16 targeting all linear modules.

Which PEFT method works best with only 200 examples for a niche format task?

Match each PEFT method to the advantage that defines it.

What does the target_modules list in a PEFT config control?

What does rank r in LoRA count?

What does PEFT stand for, and what core problem does it tackle?

When does a PEFT config need modules_to_save set, and what goes in it?

After calling merge_and_unload() on a PEFT model, what does the returned object look like?

In LoRA, where in the forward pass does lora_dropout apply its mask?

What role does the lora_alpha scalar play?

Which hyperparameter is most likely when loss spikes the moment warmup ends?

What are the three first things to check when LoRA training loss sits flat from step 0?

Compare LoRA r=8 vs r=64 on a 13B model on the quality vs cost curve

PremiumWhen does full fine-tuning…

Compute the LoRA trainable param count for a 4096x4096 projection at rank r=8

PremiumWhen should you fine-tune?

How does QLoRA fit 65B fine-tuning onto a single 48GB GPU?

Which statements about QLoRA are true?

PremiumWhy has prefix/prompt tuning…