2.6 Metrics and the confusion matrix

Key points

1. Precision = true positives / (true positives + false positives) = 80/100. Recall = true positives / (true positives + false negatives) = 80/200.

   What NVIDIA says (2)

   “The precision is the ratio tp / (tp + fp) where tp is the number of true positives and fp the number of false positives.”

   — cuML API: precision_recall_curve

   “The recall is the ratio tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives.”

   — cuML API: precision_recall_curve

2. A confusion matrix counts how often each true class was predicted as each class. Off-diagonal cells are mistakes.

   What NVIDIA says (1)

   “Normalizes confusion matrix over the true (rows), predicted (columns) conditions or al”

   — cuML API: confusion_matrix

3. Recall measures how many positives you catch. When misses are expensive, high recall matters most, even at some cost in precision.

   What NVIDIA says (1)

   “The recall is intuitively the ability of the classifier to find all the positive samples.”

   — cuML API: precision_recall_curve

4. MSE squares each error, so a few big errors dominate it. MAE averages absolute errors, so it is more robust to outliers.

   What NVIDIA says (2)

   “Some of those might be outliers, so MSE is not robust to their presence.”

   — A Comprehensive Overview of Regression Evaluation Metrics

   “Other than the scale, RMSE has the same properties as MSE.”

   — A Comprehensive Overview of Regression Evaluation Metrics

Key terms

• Accuracy: The share of predictions that are correct.
• Precision: Of the cases the model flagged as positive, the share that really are.
• Recall: Of the real positive cases, the share the model found.
• Confusion matrix: A table that counts each true class against each predicted class.
• MSE and RMSE: The average squared error, and its square root in the units of the target.
• Mean absolute error: The average size of errors, ignoring sign; less swayed by outliers than MSE.

Try it

• Confusion matrix lab: precision, recall and F1
• Regression metrics lab: R², MSE, RMSE and MAE

Sample question

A fraud model flags 100 transactions; 80 are fraud. There were 200 fraud cases in total. What are precision and recall?