Scribd
Upload
10 views

Model Performance Assessment

The document discusses various techniques for evaluating machine learning models, including accuracy, precision, recall, F1 score, confusion matrices, and cross-validation. It explains these performance metrics and techniques in detail to analyze a model's strengths and weaknesses.

Uploaded by

sanyengere
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
10 views

Model Performance Assessment

The document discusses various techniques for evaluating machine learning models, including accuracy, precision, recall, F1 score, confusion matrices, and cross-validation. It explains these performance metrics and techniques in detail to analyze a model's strengths and weaknesses.

Uploaded by

sanyengere
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 13

Model Performance Assessment

Model Performance

 Model evaluation is the process that uses some metrics which help us
to analyze the performance of the model.

 As we all know that model development is a multi-step process and a


check should be kept on how well the model generalizes future
predictions.

 Therefore evaluating a model plays a vital role so that we can judge


the performance of our model.

 The evaluation also helps to analyze a model’s key weaknesses.


Performance Metrics

 There are many metrics like Accuracy, Precision, Recall,


F1 score, Area under Curve, Confusion Matrix, and Mean
Square Error.

 Cross Validation is one technique that is followed during


the training phase and it is a model evaluation technique
as well.
Cross Validation and Holdout
Cross Validation is a method in which we do not use the whole dataset for training. In this
technique, some part of the dataset is reserved for testing the model.

There are many types of Cross-Validation out of which K Fold Cross Validation is mostly used.

In K Fold Cross Validation the original dataset is divided into k subsets. The subsets are known
as folds.

This is repeated k times where 1 fold is used for testing purposes.

Rest k-1 folds are used for training the model. It is seen that this technique generalizes the
model well and reduces the error rate.
Holdout

Holdout is the simplest approach. It is used in neural networks as well as in


many classifiers.

In this technique, the dataset is divided into train and test datasets.

The dataset is usually divided into ratios like 70:30 or 80:20.

Normally a large percentage of data is used for training the model and a
small portion of the dataset is used for testing the model.
Accuracy

Accuracy is defined as the ratio of the number of correct


predictions to the total number of predictions. This is the
most fundamental metric used to evaluate the model.

The formula is given by


Accuracy = (TP+TN)/(TP+TN+FP+FN)
Precision

Precision is the ratio of true positives to the summation of true


positives and false positives. It basically analyses the positive
predictions.

Precision = TP/(TP+FP)

The drawback of Precision is that it does not consider the True


Negatives and False Negatives.
Recall

Recall is the ratio of true positives to the summation of true


positives and false negatives. It basically analyses the number of
correct positive samples.

Recall = TP/(TP+FN)

The drawback of Recall is that often it leads to a higher false


positive rate.
F1 score

The F1 score is the harmonic mean of precision and recall. It is seen


that during the precision-recall trade-off if we increase the precision,
recall decreases and vice versa. The goal of the F1 score is to combine
precision and recall.

F1 score = (2×Precision×Recall)/(Precision+Recall)


Confusion Matrix
A confusion matrix is an N x N matrix where N is the number of target classes. It
represents the number of actual outputs and the predicted outputs. Some
terminologies in the matrix are as follows:

 True Positives: It is also known as TP. It is the output in which the actual and the
predicted values are YES.

 True Negatives: It is also known as TN. It is the output in which the actual and
the predicted values are NO.

 False Positives: It is also known as FP. It is the output in which the actual value is
NO but the predicted value is YES.

 False Negatives: It is also known as FN. It is the output in which the actual value
Confusion Matrix
Area Under Curve (AUC) /The Receiver Operating Characteristic(ROC) curve

The Receiver Operating Characteristic(ROC) curve is a probabilistic curve used to


highlight the model’s performance. The curve has two parameters:
 TPR: It stands for True positive rate. It basically follows the formula of Recall.

 FPR: It stands for False Positive rate. It is defined as the ratio of False positives to the
summation of false positives and True negatives.

This curve is useful as it helps us to determine the model’s capacity to distinguish between
different classes.

A model is considered good if the AUC score is greater than 0.5 and approaches 1. A poor
model has an AUC score of 0.
AUC or ROC

Area under the ROC Curve (AUC) measures how much better a machine learning model predicts classification
versus a random luck model.

You might also like