What is an ensemble method?

Ensemble methods combine multiple models to produce a stronger predictor. Key techniques: Bagging (Bootstrap Aggregating) — train multiple models on bootstrap samples and average predictions (e.g., Random Forest); reduces variance. Boosting — train models sequentially, each correcting the errors of the previous (e.g., AdaBoost, Gradient Boosting, XGBoost); reduces bias and variance. Stacking — train multiple base models and a meta-model to combine their predictions. Ensembles consistently win data science competitions and are a reliable way to improve model performance.