Module-1: Random Variables and Processes (BEC402)

1.1 Probability
Laplace also said, "Probability theory is nothing but common sense reduced to calculation."
Probability has grown to be one of the most essential mathematical tools applied in diverse fields like economics,
commerce, physical sciences, biological sciences and engineering. It is particularly important for solving practical
electrical-engineering problems in communication, signal processing and computers.
Probabilistic models are established from observation of a random phenomenon. While probability is concerned
with analysis of a random phenomenon, statistics help in building such models from data.
Many of the physical quantities are random in the sense that these quantities cannot be predicted with certainty and
can be described in terms of probabilistic models only.
For example,
 The outcome of the tossing of a coin cannot be predicted with certainty. Thus the outcome of tossing a coin is
random.
 The number of ones and zeros in a packet of binary data arriving through a communication channel cannot be
precisely predicted is random.
 The ubiquitous noise corrupting the signal during acquisition, storage and transmission can be modeled only
through statistical analysis. How to Interpret Probability Mathematically, the probability that an event will occur is
expressed as a number between 0 and 1.
Notationally, the probability of event A is represented by P (A).
 If P (A) equals zero, event A will almost definitely not occur.
 If P (A) is close to zero, there is only a small chance that event A will occur.
 If P (A) equals 0.5, there is a 50-50 chance that event A will occur.
 If P(A) is close to one, there is a strong chance that event A will occur.
 If P(A) equals one, event A will almost definitely occur. In a statistical experiment, the sum of probabilities for
all possible outcomes is equal to one. This means, for example, that if an experiment can have three possible
outcomes (A, B, and C), then P(A) + P(B) + P(C) = 1.

1.2 BASICS IN PROBABILITY (OR) TERMINOLOGY IN PROBABILITY

1.2.1 Outcome The outcome is an end result of an experiment.
Examples: • Getting Head or Trail in tossing a coin.
• Getting 1, 2, 3, 4, 5, 6 in throwing a dice.
1.2.2 Trail: It is the single performance of random experiment.
Example: • One attempt of rolling dice
• One attempt of tossing a coin
1.2.3 Random experiment: An experiment whose outcome s are not known in advance.
Example: • Tossing a coin.
• Rolling a dice.
• Measuring a noise voltage at the terminals of the system.
1.2.4 Random event: A random event is an outcome or set of outcomes of an random experiment that share a
common attribute.
Example: • In a rolling a die getting even number or odd number are called as random event.
i.e., Total outcome S = {1, 2, 3, 4, 5, 6}
Getting even number Ae = {2, 4, 6}
Getting odd number Ao = {1, 3, 5}
1.2.5 Mutually exclusive event: The two events A and B are said to be mutually exclusive, If they have no
common element.
Example: In a rolling a dice the total outcome S = {1, 2, 3, 4, 5, 6}
Getting even number Ae = {2, 4, 6}
Getting odd number Ao = {1, 3, 5}
Getting numbers less than 4, i.e., A4 = {1, 2, 3}
∴ Ae ∩ Ao = φ Ae and Ao −→ are mutually exclusive event. Ae ∩ A4 −→ are not mutually exclusive event.
EASWARA. M, CBIT, KOLAR 1
Module-1: Random Variables and Processes (BEC402)
1.2.6 Sample space: The set of possible outcomes of an random experiment is called sample space.
Example: In tossing a coin, sample space is {H, T}
• Sample space in rolling a dice, S = {1, 2, 3, 4, 5, 6}
• The sample space of a random experiment in which a dice and coin are tossed.
S = {(H, 1),(H, 2),(H, 3),(H, 4),(H, 5),(H, 6), (T, 1),(T, 2),(T, 3),(T, 4),(T, 5),(T, 6)}
• Sample space of tossing two coins: S = {HH, HT, TH, TT}
• Sample space of two dice rolled.
S = {(1, 1),(1, 2),(1, 3),(1, 4),(1, 5),(1, 6),
(2, 1),(2, 2),(2, 3),(2, 4),(2, 5),(2, 6)........
(6, 1),(6, 2),(6, 3),(6, 4),(6, 5),(6, 6)}

1.3 DEFINITION OF PROBABILITY

Mostly used types are 1. Relative frequency approach → Experiment
2. General approach → Theory
1.3.1) Relative frequency approach
It is based on experimentation or practical. Suppose if the random experiment is performed ‘n’ number of times
then event A has occurred nA times then probability of event A can be written as

Favorable events (nA)

P(A) =
Total no. of possible events (n)

It is also known as a posteriori probability, i.e., the probability determines after the event.

Consider two events A and B of the random experiment. Suppose we conduct ‘n’ independent trails of this
experiment and events A and B occurs in n(A) and n(B) trails vice versa.
When A and B are mutually exclusive event
∴ P(A ∪ B) = P(A + B) = P(A) + P(B)
When they are not mutually exclusive
P(A ∪ B) = P(A) + P(B) − P(A ∩ B) P(A + B) = P(A) + P(B) − P(AB)

Example:1) An experiment is repeated number of times as shown in below. Find the probability of each event.
Random Experiment Getting Head
1 1
10 6
100 50
Solution: Relative frequency:

Example 2): Suppose a coin is flipped 3 times. What is the probability of getting two tails and one head?
Solution: For this experiment, the sample space consists of 8 sample points.
S = {TTT, TTH, THT, THH, HTT, HTH, HHT, HHH}
Each sample point is equally likely to occur, so the probability of getting any particular sample point is 1/8. The
event "getting two tails and one head" consists of the following subset of the sample space.
A = {TTH, THT, HTT}
The probability of Event A is the sum of the probabilities of the sample points in A.
Therefore, P(A) = 1/8 + 1/8 + 1/8 = 3/8

1.3.2) General approach:

It is based on the axioms of theorems. In general, probability denotes the possibility of the outcome of any random
event.
EASWARA. M, CBIT, KOLAR 2
Module-1: Random Variables and Processes (BEC402)
Outcome: The end result of an experiment. For example, if the experiment consists of throwing a die, the outcome
would be anyone of the six faces, F1, ........, F6.
Random experiment: An experiment whose outcomes are not known in advance. (e.g. tossing a coin, throwing a
die, measuring the noise voltage at the terminals of a resistor etc.).
Sample space (S): The set of all possible outcomes of an experiment. See fig. 1
An event : A sample point or the set of sample points in the sample space S.
Let us consider ’S’ be the sample space consisting all possible outcomes an experiment. The events A, B, C... are
subsets of sample space. The function P(.) defines which associates with event ‘A’ is a real number called
probability of A. This function P(.) has to satisfies the following axioms.

Fig.1 Illustration of the relationship between sample space, events and probability.

A probability system has the following properties (axioms):

1 P(S) = 1 (Certainty or normalization: For sure or certain events)
2 0 ≤ P(A) ≤ 1 (Non-negativity: For every event ‘A’)
3 Additivity: If A and B are two mutually exclusive events, then
P(A ∪ B) = P(A + B) = P(A) + P(B)
Derived axioms:

2. If A and B are not two mutually exclusive events, P(A ∪ B) = P(A) + P(B) - P(A ∩ B)
3. If A1, A2, A3,…. Am, are mutually exclusive events, then P(A1) + P(A2)+ ….+ P(Am)=1
Example: 2). If two coins tossed simultaneously, determine the probability of obtaining exactly two heads.
Solution: Number of sample points = 2 × 2 = 4
S = {(T, T),(T, H),(H, T),(H, H)}; P(getting two heads) = 1/ 4
Venn Diagrams:
Pictorial representations of sets represented by closed figures are called set diagrams or Venn diagrams.
Venn diagrams are used to illustrate various operations like union, intersection and difference.
1.4 CONDITIONAL PROBABILITY
Suppose in a random experiment or signal that has been characterized by two random variables, A and B, which are
not independent. Then knowing the value of one random variable, say A, would influence the values observed for
the other random variable B.
Conditional probability of B with respect to A: It is the probability of the event B, under the condition that the
event A happens. That means, conditional probability represents the probability of B occurring, given that A has
already occurred. The conditional probability P(B|A) can be written in terms of the joint probability P(AB) and the
probability of the event P(A).

P(B|A) means "Event B, given Event A = the probability of occurrence of B when A has already occurred .

EASWARA. M, CBIT, KOLAR 3

Module-1: Random Variables and Processes (BEC402)
The formula for the Conditional Probability of an event can be derived as follows:

.
P(A|B) represents the probability of occurrence of A given B has occurred.
N(A ∩ B) is the number of elements common to both A and B.
N(B) is the number of elements in B, and it cannot be equal to zero.
Let N represent the total number of elements in the sample space.
P(A|B) = P(A ∩ B)/P(B)
Therefore, P(A ∩ B) = P(B) P(A|B) if P(B) ≠ 0
= P(A) P(B|A) if P(A) ≠ 0
Similarly, the probability of occurrence of B when A has already occurred is given by,

P(B|A) = P(B ∩ A)/P(A)

Example: 3) Let A and B are events of a experiment such that P(A) = 3/10, P(B) = ½ and P(B|A) = ⅖. Find the
value of (i) P(A ∩ B) (ii) P(A|B) (iii) P(A U B)

Solution: We know that P(A|B) = P(A ∩ B)/P(B) ⇒ P(A ∩ B) = P(A|B).P(B)

(i) P(A ∩ B) = P(B|A).P(A) = ⅖ × 3/10 = 3/25
(ii) P(A|B) = P(A ∩ B)/P(B) = (3/25) ÷ (½) = 6/25
(iii) P(A U B) = P(A) + P(B) – P(A ∩ B) = 3/10 + ½ – 3/25 = 17/25.
Example: 4) A family has two children. It is known that at least one of the children is a girl. What is the
probability that both the children are girls?
Solution: A = event of at least one girl
B = event of two girl
S={gg, gb, bg, bb},
A={gg, gb, bg}= least one of the children is a girl,
B ={gg} = probability of both the children are girls
P(B|A) = P(A ∩ B) /P(A) = (¼)/ (¾) = 1/3
Example: 5) An urn contains 6 red marbles and 4 black marbles. Two marbles are drawn without replacement from
the urn. What is the probability that both of the marbles are black?
Solution: Let A = the event that the first marble is black; and let B = the event that the second marble is black.
We know the following:
the beginning, there are 10 marbles in the urn, 4 of which are black. Therefore, P(A) = 4/10.

Therefore, based on the rule of multiplication:

P(A ∩ B) = P(A) P(B|A)
P(A ∩ B) = (4/10) * (3/9) = 12/90 = 2/15

1.5 Random Variables

Suppose that to each sample point of a sample space we assign a number (or variable). Then a function defined on
the sample space is called a random variable (or stochastic variable). It is usually denoted by a capital letter such
as X , Y, etc., and the values assumed by are denoted by lower case letters with subscripts such as x1, x2, y1, y2
etc.

EASWARA. M, CBIT, KOLAR 4

Module-1: Random Variables and Processes (BEC402)
Example: Let us consider a random experiment is tossing three coins, there are eight possible outcomes (23 = two
facesthree coins) of this experiment. The sample space can be written as,

Here, S denotes a sample space, X denotes a random variable and the condition is number of heads.

Types of random variables

1. Discrete random variable: If the random variable takes finite set of discrete values then it is called discrete
random variable. Ex: In tossing three coins, a random variable ‘X’ takes 0; 1; 2 and 3 values.
2. Continuous random variable: If the variable takes infinite set of values then it is called continuous random
variable. Ex: Temperature in a day, voltage fluctuation in a circuit, etc,

1.5.1 Probability Distribution Function :FX(x) of R.V

The cumulative distribution function of RV X, is denoted as FX(x). This is the probability that the random variable
X takes any value less than or equal to x. The cumulative distribution function (CDF) is written as

where x is any real number, i.e., -∞ < x< ∞

Properties of Distribution Function: FX(x)

1) FX(x) is bounded between zero and one
2) Fx (x) is monotonically non-decreasing function of x,

1.5.2 Probability density function - fX(x)

Defined as the derivative of the distribution function is the probability density function

Probability Distribution Function (CDF) may be recovered from the density function fX(x) by integration

EASWARA. M, CBIT, KOLAR 5

Module-1: Random Variables and Processes (BEC402)
Properties of fX(x)

Uniform Distribution Function: A random variable X is said to be uniformly distributed over the interval (a,b)
if its PDF is

Fig. 2 The uniform distribution (a) The probability density function (b) The distribution function
As PDF is non-decreasing and FX (∞) = 1.This implies that the total area under the curve of the density function
is unity.

Fig. 2 illustrates PDF for normal distribution function.

Example: 6) Find the PDF and CDF of a random experiment in which three coins are tossed and condition to get
random variable is getting head.
Solution: Sample space S = {HHH; HHT; HTH; HTT; THH; THT; TTH; TTT}
Random variable X = {x1; x2; x3; x4; x5; x6; x7; x8} No. of Heads (Condition) = {3 2 2 1 2 1 1 0}
Apply the condition to random variable X, getting head X = {0 1 2 3}
The probability density function (PDF) is the probability of random variable

EASWARA. M, CBIT, KOLAR 6

Module-1: Random Variables and Processes (BEC402)
The expression for probability density function is

1.7 Several Random Variables

Consider two random variables X and Y. We define the joint distribution function FX,Y (x,y) as the probability that
the random variable X is less than or equal to a specified value x and that the random variable Y is less than or
equal to a specified value y.

The joint distribution function FX ,Y (x,y) is a monotone non-decreasing function of both x and y. Hence, the joint
probability density function fx,Y (x,y) is always nonnegative. Also the total volume under the graph of a joint
probability density function must be unity, as shown by

The conditional distribution of Y given X is given by

If X and Y are independently distributed then P(Y= y, X= x) = P(Y= y) P(X= x) the joint distribution equals the
product of the marginal distributions.

EASWARA. M, CBIT, KOLAR 7

Module-1: Random Variables and Processes (BEC402)

and CDF

EASWARA. M, CBIT, KOLAR 8

Module-1: Random Variables and Processes (BEC402)

EASWARA. M, CBIT, KOLAR 9

Module-1: Random Variables and Processes (BEC402)

1.8 STATSTICAL AVERAGES

It means that possible ways of study of average behavior of random experiments.

Function of a random variable

Let X denote a random variable, and let g(X) denote a real-valued function defined on the real line. To find the
expected value of the random variable X,

Indeed, this Eq. may be viewed as generalizing the concept of expected value to an arbitrary function g(X) of
a random variable X.

1.9 MOMENTS
Moments are parameters (special case of g(X) = Xn) that are important in the characterization of probability
distributions and probability density functions.
Two types: (1) Moment about origin (2) Moment about mean

(1) Moment about origin:

EASWARA. M, CBIT, KOLAR 10

Module-1: Random Variables and Processes (BEC402)

(2) Moment about mean or central moment

1.9.1 CHARACTERISTIC FUNCTION

The characteristic function of a random variable X is an important statistical average that can be expressed as the
expected value of the function ejuX. Where, j = √-1.
If X has a continuous probability density function fX(x) then the characteristic function is

Invert the characteristic function to find the pdf

EASWARA. M, CBIT, KOLAR 11

Module-1: Random Variables and Processes (BEC402)

1.9.2 JOINT MOMENTS OF RANDOM VARIABLES

For two random variables X and Y the joint expectation is defined as

The correlation between random variables X and Y measured by the covariance, is given by

Let σx2 and σY2 denote the variances of X and Y, respectively. Then the covariance of X and Y, normalized with
respect to σX σY,is called the correlation coefficient of X and Y:

The two random variables X and Y are uncorrelated if and only if their covariance is zero, that is, if and only if
cov[XY] = 0
Also they are orthogonal if and only if their correlation is zero, that is, if and only if
E[XY] = 0
Moments of a Bernouli Random Variable
A Bernoulli trial is an experiment that has two possible outcomes, a success and a failure. Consider the coin-
tossing experiment where the probability of a head is p. Let X be a random variable that takes the value 0 if the
result is a tail and 1 if it is a head. We say that X is a Bernoulli random variable.
EASWARA. M, CBIT, KOLAR 12
Module-1: Random Variables and Processes (BEC402)
The probability mass function of a Bernoulli random variable is

1.10 RANDOM PROCESSES

A basic concern in the statistical analysis of communication systems is the characterization of random signals such
as voice signals, television signals, computer data, and electrical noise.
These random signals have two properties.
1) the signals are functions of time, defined on some observation interval.
2) the signals are random in the sense that before conducting an experiment, it is not possible to describe exactly
the waveforms that will be observed.
Accordingly, in describing random signals, we find that each sample point in our sample space is a function of
time. The sample space or ensemble comprised of functions of time is called a random or stochastic process.
Suppose that we assign to each sample point s a function of time in accordance with the rule:
X (t,s) -T≤ t ≤ T
where 2T is the total observation interval. For a fixed sample point sj,the graph of the function X( t,Sj) versus time t
is called a realization or sample function of the random process. To simplify the notation, we denote this sample
function as
xj(t) = X (t,sj )
Figure illustrates a set of sample functions {xj(t)| j= 1, 2, ...,n}. From this figure, we note that for a fixed time tk
inside the observation interval, the set of numbers
{x1(tk ), x2(tk )…… xn (tk }= { X(tk, S1 ), X(tk, S2), • • • X(tk,Sn)}

EASWARA. M, CBIT, KOLAR 13

Module-1: Random Variables and Processes (BEC402)

1.11 MEAN, AUTO-COVARIANCE, AND AUTO-CORRELATION FUNCTIONS

Mean, auto-covariance, and auto-correlation functions are statistical averages are used to describe the random
variable.
Mean function: (Stationary of first order) - Mean of a random process is a constant.

mX(t) is a function of time. It specifies the average behavior of X(t) over time. E denotes statistical expectation
operator. Expectation provides a description of the random variable in terms of a few parameters instead of
specifying the entire distribution function or the density function. It is far easier to estimate the expectation {E} of
a random variable from data than to estimate its distribution.
Variance function: The variance of a random variable is an estimate of the spread (dispersion) of the probability
distribution about the mean. The variance is also known as the standard deviation. If the values tend to be
concentrated near the mean, the variance is small; if the values tend to be distributed far from the mean, then the
variance is large (see fig. (a)). For discrete random variables, the variance, , is given by the expectation of the
squared distance of each outcome from the mean value of the distribution.

Fig.(a) Illustration of variance for small and large values. (b) Autocorrelation function of fluctuating random process.

Auto-correlation function (ACS):

Autocorrelation is the correlation of a function with a shifted version of itself. For energy signals:

ACF is a second order random process depends only on time difference: t2 - t1.
RX(t1, t2) is defined as the correlation between the two time samples Xt1 and Xt2 , then

RX(t1, t2) = E[Xt2 ][Xt1]

Autocorrelation function of fluctuating random process for small and large is shown in the fig. 5(b). Random
signals with different frequency content and their autocorrelations are shown in the fig.6.
Properties of ACS
1. R(τ) is a even function of τ. i.e. R(τ) = R(−τ)
2. RX (t1, t2) depends on both t1 and t2.
3. RX (τ) is maximum at τ = 0. It is equal to Mean Squared Value of the process.
4. If a signal is time shifted, its autocorrelation does not change.
5. If two random processes x(t) and y(t) are uncorrelated, then the autocorrelation of the sum x(t) = s(t)+w(t)
is equal to the sum of the autocorrelations of s(t) and w(t):
Rxx( t )= Rss ( t ) + Rww( t )
6. Power of a Wide-Sense Stationary Process The second moment or mean-square value of a real-valued
random process is given by

EASWARA. M, CBIT, KOLAR 14

Module-1: Random Variables and Processes (BEC402)

Fig. Random signals with different frequency content and their autocorrelations

Auto-covariance function:
Auto-covariance is similar to autocorrelation. Aut-covariance is the autocorrelation of the time-varying part of a
signal. Cx(t1,t2) is defined as the covariance between the two time samples Xt1 and Xt2

The variance of X(t) can be obtained as

var(Xt) is a function of time and is always non-negative.

Cross-correlation function:
Consider random processes X(t) and Y (t), then their cross correlation is defined as RX,Y (t1, t2) = E[Xt1 Yt2]
If RX,Y (t1, t2) = 0, for all t1 and t2, processes X(t) and Y (t) are orthogonal. Unlike the auto-correlation function, the
cross-correlation function is not necessarily symmetric.
RX,Y (t1, t2) ≠ RX,Y (t2, t1)
Processes X(t) and Y (t) are uncorrelated, if CX,Y (t1,t2) = 0 for all t1 and t2.
Properties of cross correlation function
1. Rxy ( t) is always a real valued function which may be positive or negative.
2. Rxy ( t ) may not necessarily have a maximum at t= 0 nor Rxy ( t ) an even function.
3. Rxy (- t ) = R yx( t )
4. |Rxy ( t)|2 ≤ Rxx( 0 ) R yy ( 0 )
5. |Rxy ( t )| =1/2 [Rxx( 0 ) +R yy ( 0 )]
6. When Rxy ( t ) = 0 , x(t) and y(t) are said to be ‘uncorrelated’ (assuming they have zeros mean.)

Ergodicity: If all of the sample functions of a random process have the same statistical
properties the random process is said to be ergodic. The most important consequence of
ergodicity is that ensemble moments can be replaced by time moments.

1.12 GAUSSIAN PROCESS AND GAUSSIAN DISTRIBUTION

Gaussian distributions N (µ, Σ) Gaussian processes GP(m(x), k(x, x’))

Distribution over vectors. Distribution over functions.
Fully specified by a mean and covariance. Fully specified by a mean function and covariance function.
The position of the random variable in the The argument of the random function plays the role of the index
vector plays the role of the index.
Definition: Gaussian random variable Y as a linear functional of X(t) as a Gaussian distribution if its probability
density function has the form

EASWARA. M, CBIT, KOLAR 15

Module-1: Random Variables and Processes (BEC402)
where µY is the mean and σ2Y is the variance of the random variable Y. A plot of this probability density function is
given in Fig.

For the special case when the Gaussian random variable Y is normalized to have a mean = zero and a variance =1,
as shown by

A Gaussian process has two main virtues.

1) Gaussian process has many properties that make analytic results possible;
2) Random processes produced by physical phenomena are often such that a Gaussian model is appropriate

CENTRAL LIMIT THEOREM

The central limit theorem provides the mathematical justification for using a Gaussian process as a model for a
large number of different physical phenomena in which the observed random variable, at a particular instant of
time, is the result of a large number of individual random events.
Ex: Distributions of Yi based on N = 1, N = 10 , N = 20 and N = 70

The central limit theorem states that the probability distribution of VN approaches a normalized Gaussian
distribution, N"(0,1) in the limit as N approaches infinity.

1.13 PROPERTIES OF A GAUSSIAN PROCESS

PROPERTY 1: If a Gaussian process X(t) is applied to a LTI filter, then the random process Y(t) developed at the
output of the filter is also Gaussian.
We assume that X(t) is a Gaussian process. The random processes Y(t) and X(t) are related by the convolution
integral.

EASWARA. M, CBIT, KOLAR 16

Module-1: Random Variables and Processes (BEC402)

PROPERTY 2: The mean and autocorrelation functions completely characterize a Gaussian random process.
Consider the set of random variables or samples X(t1), X(t2 ), X( tn), obtained by observing a random process X(t) at
times t1, t2,….tn., If the process X(t) is Gaussian, then this set of random variables is jointly Gaussian for any n, with
their n-fold joint probability density function being completely determined by specifying the set of means

and the set of auto-covariance functions

PROPERTY 3: Gaussian wide-sense stationary (WSS) processes are stationary in the strict sense.

PROPERTY 4: If the random variables X(t1), X(t2 ), X(tn), obtained by sampling a Gaussian process X( t) at
times t1 t2, ..tm are uncorrelated, that is,

then these random variables are statistically independent.

EASWARA. M, CBIT, KOLAR 17