2.5 Statistics for evaluating pipelines

Key points

1. R-squared is the share of variance in the target that a model explains. A high value alone does not show the model is unbiased or general.

   What NVIDIA says (1)

   “R² does not give any measure of bias, so you can have an overfitted (highly biased) model with a high value of R².”

   — A Comprehensive Overview of Regression Evaluation Metrics

2. RMSE (root mean squared error) is on the target's scale. But it is not the average error, because errors are squared first.

   What NVIDIA says (1)

   “an RMSE of 10 does not actually mean you are off by 10 units on average.”

   — A Comprehensive Overview of Regression Evaluation Metrics

3. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a text overlap metric. Higher ROUGE means the summary shares more with the reference.

   What NVIDIA says (2)

   “Measures overlap between machine-generated and human-generated summaries.”

   — Mastering LLM Techniques: Evaluation

   “The ROUGE score ranges between 0 and 1, with higher scores indicating higher similarity.”

   — Mastering LLM Techniques: Evaluation

Key terms

• Mean absolute error: The average size of prediction errors, which treats all errors equally.
• Root mean squared error: The square root of the mean squared error, on the same scale as the target.
• R-squared: The share of variance in the target that a model explains.
• ROUGE: A text-overlap score from 0 to 1 between generated and reference summaries.

Try it

• Metrics lab: R², MSE, RMSE and MAE

Sample question

A model scores a high R-squared. Why can that still mislead you?