Scribd
Upload
42 views

Evaluating Regression Model

The document introduces the evaluation of regression models, emphasizing the importance of loss functions and evaluation metrics in assessing model performance. It outlines various evaluation metrics such as Mean Absolute Error (MAE), Mean Bias Error (MBE), and Mean Absolute Percentage Error (MAPE), detailing their calculations, advantages, and disadvantages. The choice of an appropriate loss function is crucial and depends on factors like data characteristics and the specific algorithm used.

Uploaded by

chatgpt.bdw
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
42 views

Evaluating Regression Model

The document introduces the evaluation of regression models, emphasizing the importance of loss functions and evaluation metrics in assessing model performance. It outlines various evaluation metrics such as Mean Absolute Error (MAE), Mean Bias Error (MBE), and Mean Absolute Percentage Error (MAPE), detailing their calculations, advantages, and disadvantages. The choice of an appropriate loss function is crucial and depends on factors like data characteristics and the specific algorithm used.

Uploaded by

chatgpt.bdw
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 14

Introduction to Evaluating

Regression Models
Actual Value(Y)

Line of best fit


(Regression Line)
dependent variable

Predicted Value(Y')

Error (Residuals)
E = Y - Y'

Y-intercept

independent variable

independent variable
The objective of any machine learning
model is to understand and learn patterns
from the data which can further be used to
make predictions or answer questions or
simply just understand the underlying
pattern that is otherwise not evident
candidly.

Most of the time, the learning part is


iterative.

A model learns some patterns from the


data, we test it against some new data
that the model did not encounter during
training, we see how good or how bad a
job it did, we tweak and adjust some
parameters, then we put it to test again.
The part in which we evaluate and test our
model is where the loss functions come
into play. Evaluation metric is an integral
part of regression models.

Choosing the appropriate loss function is


very crucial and what makes one desirable
depends on the data at hand. Every
function has its own properties.

There are many factors that contribute to


the appropriate choice of a loss function
like the algorithm used, outliers in the data,
whether you want the function to be
differentiable, etc.
Loss function vs Cost function
A function that calculates loss for 1 data point is
called the loss function.

A function that calculates loss for the entire data


being used is called the cost function.

Let’s have a look at the list of Top 13 Evaluation


Metrics
Top 13 Evaluation Metrics

Mean Absolute Error (MAE)


Mean Bias Error (MBE)
Relative Absolute Error (RAE)
Mean Absolute Percentage Error (MAPE)
Mean Squared Error (MSE)
Root Mean Squared Error (RMSE)
Relative Squared Error (RSE)
Normalized Root Mean Squared Error (NRMSE)
Relative Root Mean Squared Error (RRMSE)
Root Mean Squared Logarithmic Error (RMSLE)
Hyber Loss
Log Cosh Loss
Quantile Loss

Let's delve more extensively into each evaluation


metric
Mean Absolute Error
Mean absolute error, also known as L1 loss is
one of the simplest loss functions and an easy-
to-understand evaluation metric.

It is calculated by taking the absolute


difference between the predicted values and
the actual values and averaging it across the
dataset.

Mathematically speaking, it is the arithmetic


average of absolute errors.

MAE measures only the magnitude of the errors


and doesn’t concern itself with their direction.
The lower the MAE, the higher the accuracy of a
model.

Mathematically, MAE can be expressed as follows,


Pros of the MAE
All the errors are weighted on the same scale
since absolute values are taken.

It is useful if the training data has outliers as


MAE does not penalize high errors caused by
outliers.

It provides an even measure of how well the


model is performing.

Cons of the MAE


Sometimes the large errors coming from the
outliers end up being treated as the same as
low errors.

MAE follows a scale-dependent accuracy


measure where it uses the same scale as the
data being measured. Hence it cannot be used
to compare series’ using different measures.
Mean Bias Error
Bias in “Mean Bias Error” is the tendency of a
measurement process to overestimate or
underestimate the value of a parameter. Bias
has only one direction, which can be either
positive or negative.

A positive bias means the error from the data is


overestimated and a negative bias means the
error is underestimated. Mean Bias Error (MBE)
is the mean of the difference between the
predicted values and the actual values.

This evaluation metric quantifies the overall


bias and captures the average bias in the
prediction.

It is almost similar to MAE, the only difference


being the absolute value is not taken here.

This evaluation metric should be handled


carefully as the positive and negative errors can
cancel each other out.
The formula for MBE,

Pros of the MBE


MBE is a good measure if you want to check
the direction of the model (i.e. whether there is
a positive or negative bias) and rectify the
model bias.

Cons of the MBE


It is not a good measure in terms of magnitude
as the errors tend to compensate each other.

It is not highly reliable because sometimes high


individual errors produce low MBE.
Relative Absolute Error
Relative absolute error is computed by taking
the total absolute error and dividing it by the
absolute difference between the mean and the
actual value.

RAE is expressed as,

RAE measures the performance of a predictive


model and is expressed in terms of a ratio. The
value of RAE can range from zero to one.
A good model will have values close to zero, with
zero being the best value. This error shows how the
mean residual relates to the mean deviation of the
target function from its mean.

Pros of the RAE


RAE can be used to compare models where
errors are measured in different units.

In some cases, RAE is reliable as it offers


protection from outliers.

Cons of the RAE


One main drawback of RAE is that it can be
undefined if the reference forecast is equal to
the ground truth.
Mean Absolute Percentage Error

Mean absolute percentage error is calculated


by taking the difference between the actual
value and the predicted value and dividing it by
the actual value.

An absolute percentage is applied to this value


and it is averaged across the dataset.

MAPE is also known as Mean Absolute


Percentage Deviation (MAPD).

It increases linearly with an increase in error.


The smaller the MAPE, the better the model
performance.
Pros of the MAPE

MAPE is independent of the scale of the


variables since its error estimates are in terms
of percentage.

All errors are normalized on a common scale


and it is easy to understand.

As MAPE uses absolute percentage errors, the


problem of positive values and the negative
values canceling each other out is avoided.

Cons of the MAPE


One main drawback of MAPE is when the
denominator value encounters zero. We are
faced with the “division by zero” problem as it is
not defined.
For more information, please visit:

You might also like