Lesson 7.

DATA SCIENCE AND

AUTOMATION COURSE

MASTER DEGREE SMART

TECHNOLOGY ENGINEERING

Performance metrics
TEACHER
Mirko Mazzoleni
PLACE
University of Bergamo
Outline
1. Metrics

2. Precision and recall

3. Receiver Operating Characteristic (ROC) curves

2 /14
Outline
1. Metrics

2. Precision and recall

3. Receiver Operating Characteristic (ROC) curves

3 /14
Metrics
It is extremely important to use quantitative metrics for evaluating a machine learning
model

• Until now, we relied on the cost function value for regression and classification

• Other metrics can be used to better evaluate and understand the model

• For classification
 Accuracy/Precision/Recall/F1-score, ROC curves,…
• For regression
 Normalized RMSE, Normalized Mean Absolute Error (NMAE),…

4 /14
Classification case: metrics for skewed classes
Disease dichotomic classification example

Train logistic regression model ℎ 𝒙 , with 𝑦 = 1 if disease, 𝑦 = 0 otherwise.

Find that you got 1% error on test set (99% correct diagnoses)

The 𝑦 = 1 class has very few examples with

Only 0.50% of patients actually have disease
respect to the 𝑦 = 0 class

If I use a predictor that predicts always the 𝟎 class, I get 99.5% of accuracy!!

For skewed classes, the accuracy metric can be deceptive

5 /14
Outline
1. Metrics

2. Precision and recall

3. Receiver Operating Characteristic (ROC) curves

6 /14
Precision and recall
Suppose that 𝑦 = 1 in presence of a rare class that we want to detect

Precision (How much we are precise in the detection) Confusion matrix

Of all patients where we predicted 𝑦 = 1,
what fraction actually has the disease? Actual class

Predicted class
1 (p) 0 (n)
True Positive True Positive
=
# Predicted Positive True Positive + False Positive
True positive False positive
1 (Y)
(TP) (FP)
Recall (How much we are good at detecting)
Of all patients that actually have the disease, what False negative True negative
fraction did we correctly detect as having the disease? 0 (N)
(FN) (TN)
True Positive True Positive
=
# Actual Positive True Positive + False Negative

7 /14
Trading off precision and recall
Logistic regression: 0 ≤ ℎ 𝒙 ≤ 1
At different thresholds, correspond
• Predict 1 if ℎ 𝒙 ≥ 0.5 different confusion matrices!
These thresholds can
be different from 0.5!
• Predict 0 if ℎ 𝒙 < 0.5

Suppose we want to predict 𝑦 = 1 (disease) only if very confident

• Increase threshold → Higher precision, lower recall

Suppose we want to avoid missing too many cases of disease (avoid false negatives).
• Decrease threshold → Higher recall, lower precision

8 /14
F1-score
It is usually better to compare models by means of one number only. The F1 − score can
be used to combine precision and recall

Precision(P) Recall (R) Average F1 Score

Algorithm 1 0.5 0.4 0.45 0.444 The best is Algorithm 1
Algorithm 2 0.7 0.1 0.4 0.175
Algorithm 3 0.02 1.0 0.51 0.0392
Algorithm 3 predict always 𝟏 Average says not correctly
that Algorithm 3 is the best

P+R PR • P = 0 or R = 0 ⇒ F1 score = 0
Average = F1 score = 2
2 P+R
• P = 1 and R = 1 ⇒ F1 score = 1

9 /14
Summaries of the confusion matrix
Different metrics can be computed from the confusion matrix, depending on the class of
interest (https://en.wikipedia.org/wiki/Precision_and_recall)

10 /14
Outline
1. Metrics

2. Precision and recall

3. Receiver Operating Characteristic (ROC) curves

11 /14
Ranking instead of classifying
Classifiers such as logistic regression can output a probability of belonging to a class (or
something similar).

• We can use this to rank the different istances and take actions on the cases at top of
the list

• We may have a budget, so we have to target most promising individuals

• Ranking enables to use different techniques for visualizing model performance

12 /14
Ranking instead of classifying
p n

Y 0 0 p n
Instance
True class Score N 100 100 Y
1 0
description
99 100
…………… 1 0,99 N

…………… 1 0,98
…………… 0 0,96 p n
2 0
…………… 0 0,90 Y

…………… 1 0,88 N 98 100

p n
…………… 1 0,87 2 1
Y
…………… 0 0,85 98 99
N
…………… 1 0,80 p n
…………… 0 0,70 Y
6 4
Different confusion
N 94 96
matrices by changing
Adapated from [1] the threshold

13 /14
ROC curves
ROC curves are a very general way to represent and compare the performance of
different models (on a binary classification task)

Perfection Observations
• 0,0 : predict always negative
Random • 1,1 : predict always positive
True positive rate

guessing
• Diagonal line: random classifier
• Below diagonal line: worse than random classifier
• Different classifiers can be compared
• Area Under the Curve (AUC): probability that a randomly
chosen positive instance will be ranked ahead of randomly
chosen negative instance
False positive rate

14 /14