Preliminary Concepts in Numerical Analysis

“Numerical mathematics does not require a large amount of background but it does
require a good knowledge about the background.”

Question: What is the spirit of numerical analysis?

The methods to solve the mathematical problems can be categorized in two board approaches.
✓ One category has the analytic or exact methods, which obtain the exact solution of the problem.
✓ The second category has the so-called numerical methods which produce numbers that approximately represent the
solution of the problem.
In simple words, the numerical methods obtain some approximate solution of the problems, usually in the numeric form.
The category of the exact methods is very limited in regard to the real-world problems.
Therefore, the numerical methods have to play major role in providing the some reasonable solution of the problems of
practical interest in all branches of science and engineering.

“Numerical Analysis is the field of deriving/devising, analyzing and implementing the numerical methods.”
The most common approach followed by the numerical methods is the iterative approach.
According to which, choose an initial approximation or guess to the solution and apply a set of simple computational steps
to obtain a better approximation.

Repeatedly apply the same set of steps to the better approximations, ultimately obtaining a sufficiently accurate solution
and then stop the repetition.

Each course of repetition of the set of computational steps is called iteration.

As the solution obtained by a numerical method is an approximation to the exact solution, hence it has some error.
The error can be defined as the gap between the approximate solution and the exact solution.
Question:
What are sources/types of errors while finding approximate solutions?
The errors can be categorized in the following major categories in regard to their source and type:

1. Inherent Error (quite unrelated to the numerical methods)

❑ Human mistakes/blunders in measurements
❑ Modeling errors due to simplification assumptions
❑ Data uncertainty

2. Round-off Error (undesired but have to be made)

❑ Occurs due to approximation of numbers as a computer cannot represent certain numbers exactly due to its limitation
of ability with respect to size and precision.
❑ Occurs due to approximation of numbers to limit the level of precision (i.e., not using the decimal digits after a certain
place)
❑ Occurs in certain mathematical manipulations that are highly sensitive to the round-off error.
3. Absolute Error and Relative Error
Let 𝑥𝑎 represent an approximation to a certain quantity whose true (or exact) value is 𝑥𝑡 . The gap between these two
quantities is the error.
The very first approach to quantify the error is the absolute error. It is absolute value of the difference between the true
and the approximate values.
If 𝐸𝑎 denotes the absolute error then,

Absolute Error = |True value - Approximate value ∣

or
𝐸𝑎 = 𝑥𝑡 − 𝑥𝑎
Another approach to quantify the error is the relative error. It is ratio of absolute error to the magnitude of the true
value. If 𝐸𝑟 denotes the relative error then,

absolute error ∣ True value − Approximate value ∣

Relative Error = =
∣ True value ∣ ∣ True value|
or
𝑥𝑡 − 𝑥𝑎
𝐸𝑟 =
𝑥𝑡
The relative error shows how much is the absolute error in the approximation, relative to the true value.
4. Truncation Error
“The truncation error is the difference between a approximate (truncated) value and the actual value.”
It is defined as
“Truncation error is defined as the difference between the true (analytical) derivative of a function and the derivative
obtained by numerical approximation.”
For example:
The speed of light in vacuum is 2.99792458 × 108 𝑚Τ𝑠.
The truncated value up to two decimal places is 2.99 × 108 𝑚Τ𝑠.
Hence the truncation error is the difference between 2.99792458 × 108 𝑚Τ𝑠 and 2.99 × 108 𝑚Τ𝑠,
That is 0.0079245899 × 108 𝑚Τ𝑠,
or in scientific notation, it is 7.92458 × 105 𝑚Τ𝑠.

Remark: Truncation errors mainly arise when we approximate functions represented by an infinite series instead of
using the actual value.
We shall use Taylor’s and Maclaurin’s series to calculate the truncation error.
Taylor’s Theorem:
If 𝑓 is continuously differentiable function on an interval containing the points 𝑎 and 𝑥, then the value of the function
𝑓 at 𝑥 is given by
𝑓(𝑥) = 𝑃𝑛 (𝑥) + 𝑅𝑛 (𝑥)
where
𝑓 ′′ 𝑎 𝑓𝑛 𝑎
𝑃𝑛 (𝑥) =𝑓 𝑎 + 𝑓′ 𝑎 𝑥−𝑎 + 2
𝑥 − 𝑎 + ⋯+ 𝑥−𝑎 𝑛 (1)
2! 𝑛!
𝑛
𝑓 (𝑘) 𝑎
=෍ 𝑥−𝑎 𝑘
𝑘!
𝑘=0

and
𝑓 (𝑛+1) (c) 𝑛+1 .
𝑅𝑛 (𝑥) = 𝑥−𝑎
(𝑛 + 1)!
𝑐 is any quantity between 𝑎 and 𝑥. It is Lagrange’s form of remainder.
Equation (1) is called the Taylor series or Taylor’s formula
Here 𝑃𝑛 (𝑥) is called the 𝑛th Taylor polynomial for 𝑓 about 𝑎, and 𝑅𝑛 (𝑥) is called the remainder term (or truncation
error) associated with 𝑃𝑛 (𝑥).
If the remainder is omitted, the right side of Eq. (1) is the Taylor polynomial approximation to 𝑓(𝑥).
Example:
To understand the approximation of truncation error by Taylor's series,
let us take an example. Let 𝑓(𝑥) = 𝑒 𝑥 ,
clearly which is continuously differentiable function.
Taking 𝑎 = 0,
we have 𝑓(0) = 1 and 𝑓 ′ (0) = 1 = 𝑓 (𝑘) (0)
such that
𝑥 ′
𝑓 ′′ (0) 2
𝑓 (𝑛) (0)
𝑒 = 𝑓(0) + 𝑓 (0)(𝑥 − 0) + (𝑥 − 0) + ⋯ + (𝑥 − 0)𝑛 + ⋯
2! 𝑛!
Finally, we have
𝑥 𝑥𝑛 𝑥 𝑛+1
𝑒𝑥 = 1 + 𝑥 + + ⋯+ + +⋯
2! 𝑛! (𝑛 + 1)!
If we approximate the function up to (𝑛 + 1) terms, then truncation error is the remaining terms from (𝑛 + 1) onwards.

Approximated Function Truncation Error

𝑥 𝑥𝑛 𝑥 𝑛+1
𝑒𝑥 = 1 + 𝑥 + + ⋯+ +
2! 𝑛! (𝑛 + 1)!
Actual Mathematical Formulation
Frequently Asked Questions on Truncation Errors

Q. What is meant by the truncation error?

The difference between the true value of the function and approximated value evaluated for the function is known as
truncation error.

Q. What is the difference between truncation error and round-off error?

A round-off error arises when a given numeral is rounded to the given number of digits, whereas a truncation error
arises as a result of converting infinite series into finite ones.

Q. How the truncation error be minimized?

Truncation errors can be minimized by taking more numbers of higher-order terms.

Q. How the truncation error is calculated?

The remainder Rn of the Taylor’s series is regarded as the truncation error.
More Basic Concepts
Ill-conditioned problems:
A problem is well-conditioned if a small perturbation of the input data results in a small change in the solution. If this is
not the case then the problem is said to be ill-conditioned.
Example:
Consider the differential equation
𝑦(𝑡)
ሷ − 10𝑦(𝑡)
ሶ − 11𝑦 𝑡 = 0

and suppose we seek a solution, subject to some given initial conditions, for 𝑡 ∈ 0, 𝑇 , where 𝑇 > 0.
The general solution is 𝑦 𝑡 = 𝐴𝑒 11𝑡 + 𝐵𝑒 −𝑡 where 𝐴 and 𝐵 are arbitrary constants.
Solution
❑ Suppose we are given initial conditions 𝑦 0 = 1, 𝑦ሶ 0 = −1 then it follows that 𝐴 = 0 and B = 1. Therefore the
particular solution is
𝑦 𝑡 = 𝑒 −𝑡

❑ Now consider a small change to the initial conditions. Suppose that

𝑦 0 = 1, 𝑦ሶ 0 = −0.999
Preliminary Concepts in Numerical Analysis
Then we find that
𝐴 + 𝐵 = 1 and
11𝐴 − 𝐵 = −0.999.
Adding these two equations
gives 12A = 0.001 and therefore

11.999
𝐵 =1−𝐴 =
12

0.001 11𝑡 11.999 −𝑡

y(𝑡) = e + e
12 12

Ill-conditioned problems like the above example, where the solution changes significantly with only small change in the
data, are very hard to solve numerically.
Triangle Inequality
In mathematics, the triangle inequality states that for any triangle, the sum of the lengths of any two sides must be greater than
or equal to the length of the remaining side.

(In real life, civil engineers use the triangle inequality theorem since their area of work deals with surveying, transportation, and
urban planning. The triangle inequality theorem helps them to calculate the unknown lengths and have a rough estimate of
various dimensions.)
Mathematical Definition
Let x, y ∈ ℝ then
𝑥+𝑦 ≤ 𝑥 + 𝑦
This result is called triangle inequality.
In General form 𝑛 𝑛

෍ 𝑥𝑖 ≤ ෍ 𝑥𝑖
𝑖=1 𝑖=1
Where 𝑛 ∈ ℕ and 𝑥1 , 𝑥2, … 𝑥𝑛 ∈ ℝ
Norms
A norm is a scalar valued function from a vector space into real numbers

If 𝑥 ∈ ℝ𝑛 then norm of 𝑥 is defined as 𝑥

Where
. : ℝ𝑛 → ℝ
If it satisfies the following properties

(i) Positivity: for any vector 𝑥, ∥ 𝑥 ∥≥ 0; and ∥ 𝑥 ∥= 0 if and only if 𝑥 = 0,

(ii) Triangle inequality: for any vectors 𝑥 and 𝑦, ∥ 𝑥 + 𝑦 ∥≤∥ 𝑥 ∥+∥ 𝑦 ∥ .

(iii) Homogeneity; for any scalar 𝛼 and vector 𝑥, ∥ 𝛼𝑥 ∥= |𝛼| ∥ 𝑥 ∥ .

The most commonly used vector norms belong to the family of 𝑝-norms, or 𝓁𝑝 -norms, which are defined by

𝑛 1/𝑝
𝑝
∥ 𝐱 ∥𝑝 = ෍ 𝑥𝑖 .
𝑖=1
It can be shown that for any 𝑝 > 0, ∥⋅∥𝑝 defines a vector norm. The following 𝑝-norms are of particular interest:
𝑝 = 1 : The 𝓁1 -norm
∥ 𝐱 ∥1 = 𝑥1 + 𝑥2 + ⋯ + 𝑥𝑛
𝑝 = 2 : The 𝓁2 -norm or Euclidean norm
∥ 𝐱 ∥2 = 𝑥12 + 𝑥22 + ⋯ + 𝑥𝑛2 = 𝐱𝑇 𝐱

𝑝 = ∞: The 𝓁∞ − norm
∥ 𝐱 ∥∞ = max 𝑥𝑖
1≤𝑖≤𝑛
Examples:
L-1 Norm
1
𝑝
𝑝
∥ 𝑥 ∥𝑝 = ෍ 𝑥𝑖 → 𝑝 = 1 → ෍ |𝑥|
𝑖 𝑖
1
𝑥 = −3 →∥ 𝑥 ∥1 = |1| + | − 3| + |5| = 9
5
Euclidean Norm
1 1
𝑝 2
𝑝
∥ 𝑥 ∥𝑝 = ෍ 𝑥𝑖 → 𝑝 = 2 → ෍ |𝑥|2
𝑖 𝑖
1 1
𝑥 = −3 →∥ 𝑥 ∥2 = |1|2 + | − 3|2 + |5|2 2 = (1 + 9 + 25) = 35 = 5.91
5
Max Norm
1
𝑝 𝑝
∥ 𝑥 ∥𝑝 = ∑𝑖 𝑥𝑖 → 𝑝 = ∞ → max𝑖 𝑥𝑖

1
𝑥 = −3 →∥ 𝑥 ∥∞ = 5
5
.
Sets of Functions:
Notations:
𝐶 ℝ ; set of all functions 𝑓 such that 𝑓 is continuous on the entire real line ℝ.
ሖ
𝐶(ℝ); set of functions such that 𝑓ሖ is continuous everywhere.
ሖ ⊂ 𝐶 ℝ (set inclusion is a proper inclusion)
Note: 𝐶(ℝ)
𝐶 2 (ℝ); set of functions such that 𝑓 2 is continuous everywhere.
𝐶 𝑛 (ℝ); set of functions such that 𝑓 𝑛 is continuous everywhere.
------------
𝐶 ∞ (ℝ); set of functions for which each of the derivatives exists and continuous everywhere.
Note:
ሖ
𝐶 ∞ (ℝ) ⊂… 𝐶 𝑛 (ℝ) ⊂…. ⊂ 𝐶 2 (ℝ) ⊂ 𝐶(ℝ) ⊂𝐶 ℝ

𝐶 𝑛 𝑎, 𝑏 = sets of functions 𝑓 such that 𝑓 𝑛 exists and continuous in 𝑎, 𝑏 .

Mean Value Theorem
If 𝑓(𝑥) is
(a) Continuous on [𝑎, 𝑏]
(b) Differentiable on 𝑎, 𝑏
Then there exists at least one point 𝑐 in [𝑎, 𝑏] such that

𝑓(𝑏) − 𝑓(𝑎)
𝑓 ′ (𝑐) =
𝑏−𝑎
Example:
Apply the Mean Value Theorem to 𝑓(𝑥) = 𝑥 2 − 3 on the interval [1,3].

The content of the theorem is that because

𝑓(1) = −2 and 𝑓(3) = 6,
there must exist a number 𝑐 in the interval (1,3) satisfying
𝑓 ′ (𝑐) = (6 − (−2))/(3 − 1) = 4.
It is easy to find such a 𝑐.
Since 𝑓 ′ (𝑥) = 2𝑥, the correct 𝑐 = 2.
Example:
Let 𝑎 = −2 and 𝑏 = 4 and define 𝑓: [𝑎, 𝑏] → ℝ by
𝑓(𝑥) = 2𝑥 3 − 15𝑥 2 + 24𝑥 + 21.
Verify the mean value theorem by finding 𝜉 ∈ (−2,4) such that

𝑓(𝑏) − 𝑓(𝑎) 𝑓(4) − 𝑓(−2)

𝑓 ′ (𝜉) = =
𝑏−𝑎 4+2
Give 𝜉 to 4 decimal places.
Solution
Here we find that 𝑓(𝑎) = 𝑓(−2) = −103 and 𝑓(𝑏) = 𝑓(4) = 5 and therefore

𝑓(𝑏) − 𝑓(𝑎) 108

= = 18
𝑏−𝑎 6
Clearly 𝑓 ′ (𝑥) = 6𝑥 2 − 30𝑥 + 24 and therefore we seek 𝜉 such tat

6𝜉 2 − 30𝜉 + 24 = 18
so that
𝜉 2 − 5𝜉 + 1 = 0.
Solving the quadratic is straightforward and we find that 𝜉 = 0.2087 or 𝜉 = 4.7913, to 4 decimal places. Only one of these is
in the interval (𝑎, 𝑏) so we take 𝜉 = 0.2087.
Difference Equations
Difference equation is a relation between the differences of unknown function at one or more general values of the
independent variable.

𝑦𝑛+2 − 𝑦𝑛+1 + 𝑦𝑛 = 𝑛2
In symbolic form it can be expressed as
𝐸 2 − 𝐸 + 1 𝑦𝑛 = 𝑛2
Example:
Solve 9𝑦n+2 + 9𝑦n+1 + 2𝑦𝑛 = 0.
Solution: The given equation, in symbolic form is written as
9𝐸 2 + 9𝐸 + 2 𝑦𝑛 = 0
Thus its auxiliary equation becomes
1 2
9𝐸 2 + 9𝐸 + 2 = 0 or 𝐸 = − , −
3 3
Hence its complete solution is
𝑦𝑛 = 𝑐1 (−1/3)𝑛 + 𝑐2 (−2/3)𝑛 .
Problem:
Solve the difference equation 𝑦𝑚+3 + 16𝑦𝑚−1 = 0.
Example:
Solve the difference equation 𝑢𝑛+3 − 2𝑢𝑛+2 − 5𝑢𝑛+1 + 6𝑢𝑛 = 0

Given that 𝑢0 = 1, 𝑢1 = 2, 𝑢2 = 3.

Solution: The given equation in symbolic form is

𝐸 3 − 2𝐸 2 − 5𝐸 + 6 𝑢𝑛 = 0
and its auxiliary equation becomes
𝐸 3 − 2𝐸 2 − 5𝐸 + 6 = 0
(𝐸 − 1)(𝐸 − 3)(𝐸 + 2) = 0 or 𝐸 = 1,3, −2
Hence, the complementary function
𝑢𝑛 = 𝑐1 + 𝑐2 3𝑛 + 𝑐3 (−2)𝑛
Now using given conditions, from (3), we get
For
𝑛 = 0, 𝑢0 = 𝑐1 + 𝑐2 + 𝑐3 i.e. 𝑐1 + 𝑐2 + 𝑐3 = 1
𝑛 = 1, 𝑢1 = 𝑐1 + 3𝑐2 − 2𝑐3 i.e. 𝑐1 + 3𝑐2 − 2𝑐3 = 2
𝑛 = 2, 𝑢2 = 𝑐1 + 9𝑐2 + 4𝑐3 i.e. 𝑐1 + 9𝑐2 + 4𝑐3 = 3
Therefore,
5 3 𝑛 2
𝑢𝑛 = + 3 − (−2)𝑛
6 10 15
is the required solution of the given equation.