Unit-6

Classification and
Regression
Linear Regression-

• In Machine Learning, Linear Regression is a

supervised machine learning algorithm.
• It tries to find out the best linear relationship that
describes the data you have.
• It assumes that there exists a linear relationship
between a dependent variable and independent
variable(s).
• The value of the dependent variable of a linear
regression model is a continuous value i.e. real
numbers.
Representing Linear Regression Model-
Linear regression model represents the linear relationship between
a dependent variable and independent variable(s) via a sloped
straight line.
The sloped straight line representing the linear relationship that
fits the given data best is called as a regression line.
Types of Linear Regression-
• Simple Linear Regression

Y = β0 + β 1 X
• Y is a dependent variable.
• X is an independent variable.
• β0 and β1 are the regression coefficients.
• β0 is the intercept or the bias that fixes the offset to a
line.
• β1 is the slope or weight that specifies the factor by
which X has an impact on Y.
• Multiple Linear Regression

Y = β0 + β1X1 + β2X2 + β3X3 + …… + βnXn

• Y is a dependent variable.
• X1, X2, …., Xn are independent variables.
• β0, β1,…, βn are the regression coefficients to
be determined through regression analysis.
Types of Relationship
• Positive Linear Relationship:
When dependent variable increases on y axis as
independent variable increases on x axis
Y=a0+a1x
• Negative Linear Relationship
When dependent variable decreases on y axis as
independent variable increases on x axis
Y=-a0+a1x
Evaluation Metrics for Linear Regression

A variety of evaluation measures can be used to determine the strength

of any linear regression model. These assessment metrics often give

an indication of how well the model is producing the observed

outputs.

Mean Square Error (MSE)

Mean Absolute Error (MAE)
Root Mean Squared Error (RMSE)
Mean Absolute Percentage Error(MAPE)
R-squared(R2)
Confusion Matrix
ROC AUC Curve
Log Loss
• Mean Squared Error (MSE) is an evaluation
metric that calculates the average of the squared
differences between the actual and predicted
values for all the data points. The difference is
squared to ensure that negative and positive
differences don’t cancel each other out.
MSE=1/n∑ (y–y^)/2
• Here,
• n is the number of data points.
• y is the actual or observed value
• y^is the predicted value for the i th data point.
Mean Absolute Error (MAE)
• Mean Absolute Error is an evaluation metric used to calculate the
accuracy of a regression model. MAE measures the average
absolute difference between the predicted values and actual
values.

• MAE=1/n∑∣Y–Y^∣

• Here,
• n is the number of observations
• Y represents the actual values.
• Y^ represents the predicted values

• Lower MAE value indicates better model performance. It is not

sensitive to the outliers as we consider absolute differences
Root Mean Squared Error (RMSE)
• The square root of the residuals variance is
the Root Mean Squared Error. It describes how
well the observed data points match the expected
values, or the model’s absolute fit to the data.
• In mathematical notation, it can be expressed as

RMSE=
MSE
Mean Absolute Percentage Error(MAPE)

• MAPE is defined as the average absolute percentage

difference between predicted values and actual values.
• Also known as the mean absolute percentage
deviation (MAPD)

• Formula: 1/N Σ(|actual-predicted|/|actual|)*100

• N is the number of fitted points;
• A is the actual value;
• F is the forecast value; and
• Σ is summation notation (the absolute value is summed
for every forecasted point in time).
R Squared

R squared measures the proportion of the

variance in the dependent variables.

• R2 = 1 – (RSS/TSS)
• R2 represents the required R Squared value,
• RSS represents the residual sum of squares, and
• TSS represents the total sum of squares.
• (txt book)
Confusion Matrix
• Used in classification problem which displays
TP,TN,FP,FN counts.
• It is also used to calculate various classification
metrics such as Accuracy, Precision, Recall and
F1 score.
ROC curve
• ROC curve used in binary classification , plots
the true positive rate (TPR) against the False
positive rate (FPR) at different threshold.

• AUC measures the area under the curve.

• Higher the AUC better the model.

Log Loss
• Log Loss used in the classification problems

• It quantifies the difference between predicted

probabilities and true class labels.

• Lower log loss values indicate better

performance.
Overfitting and Underfitting
Overfitting:
• Overfitting occurs when our machine learning model tries to
cover all the data points or more than the required data
points present in the given dataset.
• Because of this, the model starts caching noise and
inaccurate values present in the dataset, and all these factors
reduce the efficiency and accuracy of the model.
Reasons for Overfitting:
• The model is too complex.
• The size of the training data.
• High variance and low bias.
Underfitting:
• In the case of underfitting, the model is not able to learn
enough from the training data, and hence it reduces the
accuracy and produces unreliable predictions.
Reasons for Underfitting
• The model is too simple, So it may be not capable to
represent the complexities in the data.
• The input features which is used to train the model is not
the adequate representations of underlying factors
influencing the target variable.
• The size of the training dataset used is not enough.
• Excessive regularization are used to prevent the overfitting,
which constraint the model to capture the data well.
• Features are not scaled.
Calculate MSE using Excel
Step 1: Enter the actual values and forecasted values in two separate
columns.
Step 2: Calculate the squared error for
each row.
Step 3: Calculate the mean squared error.
Calculate MSE by simply finding the average of the values in column D:
MSE=1/n∑ (y–y^)2
Logistic Regression
• Logistic regression is used in binary classification where
we use sigmoid function, that takes input as independent
variables and produces a probability value between 0 and
1.

• For example, we have two classes Class 0 and Class 1 if

the value of the logistic function for an input is greater
than 0.5 (threshold value) then it belongs to Class 1
otherwise it belongs to Class 0. It’s referred to as
regression because it is the extension of linear
regression but is mainly used for classification problems.
• Logistic regression predicts the output of a categorical
dependent variable. Therefore, the outcome must be a
categorical or discrete value.
• It can be either Yes or No, 0 or 1, true or False, etc. but
instead of giving the exact value as 0 and 1, it gives
the probabilistic values which lie between 0 and 1.
• In Logistic regression, instead of fitting a regression
line, we fit an “S” shaped logistic function, which
predicts two maximum values (0 or 1).
Binary And Multiclass Classification
In binary classification, the goal is to classify the input
into one of two classes or categories.
Example – On the basis of the given health conditions
of a person, we have to determine whether the person
has a certain disease or not.

In multi-class classification, the goal is to classify the

input into one of several classes or categories.
Examples of multiclass classification include: face
classification, animal species classification
Assessing in Logistic regression
Confusion Matrix and classification Report:
It consist of Four components TP,TN,FP,FN
It basically assess the model’s Accuracy, precision call, recall,
F1 sore

• ROC Curve and AUC Score:

It is assessed by calculating TPR against FPR at different
threshold values.
It summarizes overall performance of the model with higher
AUC value
Indicating better performance.
• Calibration curve:
Used to assess how well the predicted
probabilities align with the Actual probabilities.

• Residual Plot
Residual plot is used for assessing the
performance of Regression Model.
It displays the discrepancies between predicted
and actual values.
• Multiclass Classification
Binary logistic regression
• Binary logistic regression works well for binary classification problems
that have only two possible outcomes. The dependent variable can have
only two values, such as yes and no or 0 and 1.
• Even though the logistic function calculates a range of values between 0
and 1, the binary regression model rounds the answer to the closest values.
Generally, answers below 0.5 are rounded to 0, and answers above 0.5 are
rounded to 1, so that the logistic function returns a binary outcome.

Multinomial logistic regression

• Multinomial regression can analyze problems that have several possible
outcomes as long as the number of outcomes is finite. For example, it can
predict if house prices will increase by 25%, 50%, 75%, or 100% based on
population data, but it cannot predict the exact value of a house.
• Multinomial logistic regression works by mapping outcome values to
different values between 0 and 1. Since the logistic function can return a
range of continuous data, like 0.1, 0.11, 0.12, and so on, multinomial
regression also groups the output to the closest possible values
One vs One
In One-vs-One classification, for the N-class instances dataset, we have to generate
the N* (N-1)/2 binary classifier models. Using this classification approach, we split
the primary dataset into one dataset for each class opposite to every other class.
One vs Rest
In one-vs-All classification, for the N-class instances dataset, we have to generate
the N-binary classifier models. The number of class labels present in the dataset
and the number of generated binary classifiers must be the same.
• As shown in the above image, consider we
have three classes, for example, type 1 for
Green, type 2 for Blue, and type 3 for Red.
• So we have to create three classifiers here for
three respective classes.
• Classifier 1:- [Green] vs [Red, Blue]
• Classifier 2:- [Blue] vs [Green, Red]
• Classifier 3:- [Red] vs [Blue, Green]
Sums on Confusion Matrix
• TP=86
• TN=79
• FP=12
• FN=10
Accuracy=?
Precision=?
Recall=?
F1score=?
Linear Regression vs Logistic Regression

Linear Regression Logistic Regression

Linear regression is used to predict the Logistic regression is used to predict the
continuous dependent variable using a given categorical dependent variable using a given
set of independent variables. set of independent variables.

Linear regression is used for solving regression

problem. It is used for solving classification problems.

In this we predict the value of continuous

variables In this we predict values of categorical variables

In this we find best fit line. In this we find S-Curve.

Least square estimation method is used for Maximum likelihood estimation method is used
estimation of accuracy. for Estimation of accuracy.

The output must be continuous value, such as Output must be categorical value such as 0 or
price, age, etc. 1, Yes or no, etc.

It required linear relationship between It not required linear relationship.

dependent and independent variables.