gradient boosting

Gradient Boosting

An ensemble ML technique that builds models sequentially, where each model corrects the errors of the previous one — typically produces state-of-the-art accuracy on structured tabular data.

When you'd see it: Production ML models for churn prediction, fraud detection, demand forecasting, click-through rate prediction, credit scoring. XGBoost and LightGBM are the dominant implementations. When a data science team says their model 'outperformed the baseline,' gradient boosting is often what beat it.

Why it matters: Gradient boosting consistently tops benchmark performance on tabular data problems. It handles non-linear relationships, interactions, and missing values without manual feature engineering. The cost: more hyperparameters to tune than Random Forest, and higher overfitting risk if regularization isn't set carefully.

Common mistakes: Treating gradient boosting as plug-and-play without tuning. Default parameters often overfit. Key parameters: learning rate (smaller = more robust, slower), max depth (deeper = more overfitting risk), and number of trees (more trees with a small learning rate is the standard recipe).

Related Random Forest, decision tree, overfitting, AUC, confusion matrix