Topic 3 Multiple Regression

Analysis: Estimation
2
Multiple Regression Model
• Multiple Regression Model
The equation that describes how the dependent variable y is related to the independent variables x1,
x2, . . . xp and an error term is:
Classical Linear Regression Model (CLRM)
 A-1: The regression model is linear in the parameters; it may or may not be linear in the
variables Y and the Xs.
 A-2: The regressors are assumed to be fixed or nonstochastic in the sense that their values
are fixed in repeated sampling. Fixed X values or X values independent of the error term.
Hence, this means we require zero covariance between and each X variables.

 A-3: Given the values of the X variables, the expected, or mean, value of the error term is
zero. That is,

 A-4: The variance of each ui, given the values of X, is constant, or homoscedastic (homo
means equal and scedastic means variance). That is,
Classical Linear Regression Model (CLRM)
 A-5: There is no correlation between two error terms. That is, there is no autocorrelation.
Symbolically,

 A-6: There are no perfect linear relationships among the X variables. This is the
assumption of no multicollinearity. For example, relationships like are ruled out.
 A-7: The regression model is correctly specified. Alternatively, there is no specification
bias or specification error in the model used in empirical analysis. It is implicitly
assumed that the number of observations, n, is greater than the number of parameters
estimated.

Although it is not a part of the CLRM, it is assumed that the error term follows the normal
distribution with zero mean and (constant) variance . Symbolically,
 A-8:
Classical Linear Regression Model (CLRM)
 On the basis of Assumptions A-1 to A-7, it can be shown that the method of ordinary least
squares (OLS), the method most popularly used in practice, provides estimators of the
parameters of the population regression function (PRF) that have several desirable
statistical properties, such as:
1. The estimators are linear, that is, they are linear functions of the dependent variable Y.
Linear estimators are easy to understand and deal with compared to nonlinear estimators.
2. The estimators are unbiased, that is, in repeated applications of the method, on average,
the estimators are equal to their true values.
3. In the class of linear unbiased estimators, OLS estimators have minimum variance. As a
result, the true parameter values can be estimated with least possible uncertainty; an
unbiased estimator with the least variance is called an efficient estimator.

 In short, under the assumed conditions, OLS estimators are BLUE: best linear unbiased
estimators. This is the essence of the well-known Gauss–Markov theorem, which
provides a theoretical justification for the method of least squares.
Multiple Regression Analysis: Estimation
6

 Motivation for multiple regression

 Incorporate more explanatory factors into the model
 Explicitly hold fixed other factors that otherwise would be in
 Allow for more flexible functional forms

 Example: Wage equation

Multiple Regression Analysis: Estimation
7

 Example: Average test scores and per student spending

 Per student spending is likely to be correlated with average family income at a given high school because of
school financing.
 Omitting average family income in regression would lead to biased estimate of the effect of spending on
average test scores.
 In a simple regression model, effect of per student spending would partly include the effect of family
income on test scores.
Multiple Regression Analysis: Estimation
8

 Example: Family income and family consumption

 Model has two explanatory variables: inome and income squared

 Consumption is explained as a quadratic function of income
 One has to be very careful when interpreting the coefficients:
Multiple Regression Analysis: Estimation
9

 Example: CEO salary, sales and CEO tenure

 Model assumes a constant elasticity relationship between CEO salary and the sales of his or her firm.
 Model assumes a quadratic relationship between CEO salary and his or her tenure with the firm.

 Meaning of “linear” regression

 The model has to be linear in the parameters (not in the variables)
Multiple Regression Analysis: Estimation
10

 Example: Determinants of college GPA

 Interpretation
 Holding ACT fixed, another point on high school grade point average is associated with another .453 points college
grade point average
 Or: If we compare two students with the same ACT, but the hsGPA of student A is one point higher, we predict
student A to have a colGPA that is .453 higher than that of student B
 Holding high school grade point average fixed, another 10 points on ACT are associated with less than one point on
college GPA
11
Multiple Regression Equation
• Multiple Regression Equation
The equation that describes how the mean value of y is related to x1, x2,
. . . xk is:
E(y) = 0 + 1x1 + 2x2 + . . . + kxk
12 Estimated Multiple Regression Equation
• Estimated Multiple Regression Equation

= b0 + b1x1 + b2x2 + . . . + bkxk

A simple random sample is used to compute sample statistics b0, b1, b2, . . . , bk
that are used as the point estimators of the parameters b0, b1, b2, . . . , bk.
13
Estimation Process
Multiple Regression Model
Sample Data:
E(y) = 0 + 1x1 + 2x2 +. .+ kxk + e x1 x2 . . . xk y
Multiple Regression Equation . . . .
E(y) = 0 + 1x1 + 2x2 +. . .+ kxk . . . .
Unknown parameters are
b0 , b1 , b2 , . . . , bk

Estimated Multiple
Regression Equation
b0, b1, b2, . . . , bk = b0 + b1x1 + b2x2 + . . . + bkxk
provide estimates of Sample statistics are
b 0 , b 1 , b2 , . . . , b k b0, b1, b2, . . . , bk
14 Least Squares Method
 Least Squares Criterion

min

• Computation of Coefficient Values

The formulas for the regression coefficients b0, b1, b2, . . . bk involve the
use of matrix algebra. We will rely on computer software packages to
perform the calculations.
The emphasis will be on how to interpret the computer output rather
than on how to make the multiple regression computations.
15 Multiple Regression Model
• Example: Programmer Salary Survey
A software firm collected data for a sample of 20 computer programmers. A
suggestion was made that regression analysis could be used to determine if
salary was related to the years of experience and the score on the firm’s
programmer aptitude test.
The years of experience, score on the aptitude test, and corresponding annual
salary ($1000s) for a sample of 20 programmers is shown on the next slide.
16 Multiple Regression Model
Exper. Test Salary Exper. Test Salary
(Yrs.) Score ($1000s) (Yrs.) Score ($1000s)
4 78 24.0 9 88 38.0
7 100 43.0 2 73 26.6
1 86 23.7 10 75 36.2
5 82 34.3 5 81 31.6
8 86 35.8 6 74 29.0
10 84 38.0 8 87 34.0
0 75 22.2 4 79 30.1
1 80 23.1 6 94 33.9
6 83 30.0 3 70 28.2
6 91 33.0 3 89 30.0
17 Multiple Regression Model
Suppose we believe that salary (y) is related to the years of experience
(x1) and the score on the programmer aptitude test (x2) by the following
regression model:
y = 0 + 1x1 + 2x2 + 

where
y = annual salary ($1000s)
x1 = years of experience
x2 = score on programmer aptitude test
18
Solving for the Estimates of 0, 1, 2

Least Squares
Input Data Output
x1 x2 y Computer b0 =
Package b1 =
4 78 24 for Solving
7 100 43 b2 =
Multiple
. . .
Regression R2 =
. . .
3 89 30 Problems etc.
19
Solving for the Estimates of 0, 1, 2
• Regression Equation Output

Predictor Coef SE Coef T p

Constant 3.17394 6.15607 0.5156 0.61279
Experience 1.4039 0.19857 7.0702 1.9E-06
Test Score 0.25089 0.07735 3.2433 0.00478
20 Estimated Regression Equation

SALARY = 3.174 + 1.404(EXPER) + 0.251(SCORE)

(Note: Predicted salary will be in thousands of dollars.)

21 Interpreting the Coefficients
• In multiple regression analysis, we interpret each regression coefficient as
follows:
bi represents an estimate of the change in y corresponding to one
unit increase in xi when all other independent variables are held
constant.
22 Interpreting the Coefficients

b1 = 1.404

Salary is expected to increase by $1,404 for each additional year of

experience (when the variable score on programmer attitude test is
held constant).
23 Interpreting the Coefficients

b2 = 0.251

Salary is expected to increase by $251 for each additional point scored on

the programmer aptitude test (when the variable years of experience is held
constant).
24 Multiple Coefficient of Determination
• Relationship Among SST, SSR, SSE

SST = SSR + SSE

where:
SST = total sum of squares
SSR = sum of squares due to regression
SSE = sum of squares due to error
25
Multiple Coefficient of Determination
• ANOVA Output

Analysis of Variance

SOURCE DF SS MS F P
Regression 2 500.3285 250.164 42.76 0.000
Residual Error 17 99.45697 5.850
Total 19 599.7855
26
Multiple Coefficient of Determination

R2 = SSR/SST

R2 = 500.3285/599.7855 = .83418
27 Adjusted Multiple Coefficient of Determination
• Adding independent variables, even ones that are not statistically significant,
causes the prediction errors to become smaller, thus reducing the sum of
squares due to error, SSE.
• Because SSR = SST – SSE, when SSE becomes smaller, SSR becomes larger,
causing R2 = SSR/SST to increase.
• The adjusted multiple coefficient of determination compensates for the number
of independent variables in the model.
28 Adjusted Multiple Coefficient of Determination

2 2 𝑛 −1
𝑅𝑎 =1−(1 − 𝑅 )
𝑛 −𝑘 −1

2 20 −1
𝑅𝑎 =1− ( 1−.834179 ) =.814671
20 −2 −1
29
Assumptions About the Error Term 
• The error  is a random variable with mean of zero.
• The variance of  , denoted by  2, is the same for all values of the
independent variables.
• The values of  are independent.
• The error  is a normally distributed random variable reflecting the deviation
between the y value and the expected value of y given by 0 + 1x1 + 2x2 + . .
+ kxk.
Components
30 of OLS Variances:

 1) The error variance

 A high error variance increases the sampling variance because there is more “noise” in the equation.
 A large error variance doesn‘t necessarily make estimates imprecise.
 The error variance does not decrease with sample size.

 2) The total sample variation in the explanatory variable

 More sample variation leads to more precise estimates.
 Total sample variation automatically increases with the sample size.
 Increasing the sample size is thus a way to get more precise estimates.
Components
31 of OLS Variances:

 3) Linear relationships among the independent variables

 Regress xj on all other independent variables (including constant)

 The R-squared of this regression will be the higher when xj can be better explained by the other
independent variables.

 The sampling variance of the slope estimator for xj will be higher when xj can be better
explained by the other independent variables.

 Under perfect multicollinearity, the variance of the slope estimator will approach infinity.