5.6 Parameters, tuning and fitting

Key points

1. Training adjusts parameters, such as tree splits or linear weights. Tuning searches hyperparameters by training many times.

   What NVIDIA says (1)

   “This is particularly important because data scientists typically run the algorithm not just once, but many times in order to tune hyperparameters (such as learning rate or tree depth) and find the best accuracy.”

   — Gradient Boosting, Decision Trees and XGBoost with CUDA

2. Underfitting is high bias: the model misses real structure. Overfitting is the opposite: it learns noise and fails on new data.

   What NVIDIA says (1)

   “Random forest “bagging” minimizes the variance and overfitting, while GBDT “boosting” minimizes the bias and underfitting.”

   — What Is XGBoost and Why Does It Matter?

3. Each tree overfits in its own way. Averaging their votes cancels much of that noise.

   What NVIDIA says (1)

   “The presence of a large number of trees also reduces the problem of overfitting, which occurs when a model incorporates too much “noise” in the training data and makes poor decisions as a result.”

   — NVIDIA Glossary: Random Forest

Key terms

• Overfitting: When a model learns noise in the training data and does poorly on new data.
• Underfitting: When a model is too simple to capture the pattern, even on training data.
• Hyperparameter: A setting chosen before training, such as tree depth.

Sample question

What is the difference between a model parameter and a hyperparameter?