What is fine-tuning a pre-trained model?

Fine-tuning is the process of taking a model pre-trained on a large dataset and continuing training on a smaller, task-specific dataset. Instead of training from scratch, you initialize the model with pre-trained weights (which encode general knowledge) and run a few more epochs with a low learning rate to adapt to the new task. Common strategies: full fine-tuning (update all weights), feature extraction (freeze pre-trained layers, only train the new head), and layer-wise learning rate decay (lower LR for earlier layers). For LLMs, LoRA (Low-Rank Adaptation) enables efficient fine-tuning with few trainable parameters.