Peft

Walk through what happens during backprop for a LoRA wrapped linear layer

If LoRA freezes the base weights, does that mean activation checkpointing on the base is unnecessary?

What does Adam's two fp32 buffers per trainable parameter imply for LoRA vs full FT on a 7B?

Estimate the trainable param count for a Llama-3 8B LoRA at r=16 targeting all linear modules.

Which PEFT method works best with only 200 examples for a niche format task?

Match each PEFT method to the advantage that defines it.

What does the target_modules list in a PEFT config control?

What does rank r in LoRA count?

What does PEFT stand for, and what core problem does it tackle?

When does a PEFT config need modules_to_save set, and what goes in it?

After calling merge_and_unload() on a PEFT model, what does the returned object look like?

In LoRA, where in the forward pass does lora_dropout apply its mask?

What role does the lora_alpha scalar play?

Compute the LoRA trainable param count for a 4096x4096 projection at rank r=8

How does QLoRA fit 65B fine-tuning onto a single 48GB GPU?

Which statements about QLoRA are true?

PremiumWhy has prefix/prompt tuning…

What is the architectural reason LoRA tends to win over prompt tuning?

Match PEFT method to the scenario it fits best

PremiumWhy is 'LoRA is…

What is the best architecture for a chatbot over a frequently updated company knowledge base?

Which of these are linear layers that 'all linear' LoRA targets in a SwiGLU transformer?

PremiumWhen does targeting all…

PremiumPick a LoRA rank…

Match LoRA rank to task type