What is gradient descent?

Gradient descent is the primary optimization algorithm used to train ML models. It iteratively updates model parameters (weights) in the direction that minimizes the loss function. The gradient (partial derivatives of the loss) tells us how to move to increase the loss; we move in the opposite direction. The learning rate controls the step size. Variants include Stochastic Gradient Descent (SGD) (one sample per update), Mini-batch GD (subset per update), and advanced optimizers like Adam and RMSProp.