What is LoRA (Low-Rank Adaptation)?

LoRA is a parameter-efficient fine-tuning (PEFT) technique that freezes the pre-trained model weights and injects trainable low-rank decomposition matrices into Transformer layers. Instead of updating the full weight matrix W ∈ ℝ^{d×k}, LoRA approximates the update as ΔW = BA where B ∈ ℝ^{d×r} and A ∈ ℝ^{r×k}, with rank r ≪ min(d,k). This reduces trainable parameters by orders of magnitude (e.g., 10,000× for large models) with minimal performance loss. LoRA is widely used for fine-tuning LLMs on consumer hardware. Variants: QLoRA (quantized LoRA), AdaLoRA (adaptive rank).