Lecture 4.

Concepts of Probability
AGEC 2001 Statistics I

Feng-An Yang1

1 Departmentof Agricultural Economics

National Taiwan University

Fall Semester

1/38
Outline

Concepts of Probability
Types of Probability
Definitions
Operations

Conditional Probability and Independence

Conditional Probability
Independence

Rules of Addition and Multiplication

Rules of Addition
Rules of Multiplication

Bayes’s Theorem

2/38
Concepts of Probability

Probability
A value between 0 and 1 inclusive that represents the likelihood a
particular event happens: 0 ≤ P(A) ≤ 1
I Classical Probability
I Empirical Probability
I Subjective Probability

3/38
Concepts of Probability

Classical Probability
Classical probability is based on the assumption that each outcome
is equally likely to occur
Number of outcomes favorable to A
P(A) = Number of possible outcomes

Example
I Consider an experiment of tossing a two-sided coin. What is
the probability of flipping tails on a coin?
P(A) = Number of outcomes favorable to A
Number of possible outcomes = 12
I Consider an experiment of rolling a pair of six-sided dice.
What is the probability of getting a 4?
P(A) = Number of outcomes favorable to A
Number of possible outcomes
3
= 36 1
= 12
(1, 3), (2, 2), (3, 1)

4/38
Concepts of Probability

Empirical Probability
Empirical probability is based on the number of times an event
occurs as a proportion of a known number of trials
Number of times the event A occurs
P(A) = Number of observations

I Last semester 100 students took the statistics class and 20 of

them earned an A. What is the probability for a student to
receive an A?
P(A) = Number of times the event A occurs
Number of observations
20
= 100

Law of Large Numbers

Over a large number of trials, the empirical probability of an event
will approach its true probability

5/38
Concepts of Probability

Example
Number of trials Number of Heads Relative Frequency of Heads
1 0 0.00
10 3 0.30
50 26 0.52
100 52 0.52
500 236 0.472
1,000 494 0.494
10,000 5,027 0.5027

I If we flip a coin 50 times, the probability of getting a head is

26/50=0.52
I According to the Law of Large Numbers, the empirical
probability of getting a head will approach the true probability
0.5

6/38
Concepts of Probability

Subjective Probability
Subjective probability is based on past experience. It is the
likelihood (probability) of a particular event happening that is
assigned by an individual based on whatever information is
available
I Imagine you observe that swallows often fly low just before it
rains
I Based on your experience, you might assign a subjective
probability to the likelihood of rain whenever you see swallows
flying low
I When swallows fly low, there’s an 80% chance of rain, even
though this isn’t based on precise meteorological data

7/38
Concepts of Probability

Pros and Cons

I Classical Probability
I Conceptually simple for many applications
I Does not apply if there are infinite outcomes
I Does not apply if outcomes are not equally likely
I Empirical Probability
I Covers more cases than classical
I The empirical probability might not agree with classical
probability
I How large n do we need to have a good approximation to the
true probability?
I Subjective Probability
I Applicable in cases where other ways do not apply
I The probability can vary across individuals or vary across time
for the same individual

8/38
Concepts of Probability
Definitions

Experiment
A process that lead to the occurrence of one and only one of
several possible results

Outcome or sample point

A particular result of an experiment (an element of the sample
space)

Sample space
The set, S, of all possible outcomes of a particular experiment is
called the sample space for the experiment

Event
An event, A, is any collection of possible outcomes of an
experiment, that is, any subset of S (including S itself)

9/38
Concepts of Probability
Definitions

Experiment
The action of tossing a six-sided die

Outcome or sample point

The result of a single toss of the die, which could be any one of
the numbers {1, 2, 3, 4, 5, 6}.

Sample space
The set of all possible outcomes when tossing the die. In this case,
the sample space is {1, 2, 3, 4, 5, 6}.

Event
An event could be “rolling an even number,” which would include
the outcomes {2, 4, 6}, or it could be “rolling a 3,” which would
be the event {3}.

10/38
Concepts of Probability
Operations

Union
The union of A and B, written A ∪ B, is the set of elements that
belong to either A or B or both:

A ∪ B = {x : x ∈ A or x ∈ B}

Intersection
The intersection of A and B, written A ∩ B, is the set of elements
that belong to either A and B:

A ∩ B = {x : x ∈ A and x ∈ B}

Complementation
The complement of A, written Ac , is the set of all elements that
are not in A:
Ac = {x : x ∈ / A}
11/38
Concepts of Probability
Special events

Mutually Exclusive Events

Two or more events are mutually exclusive (or disjoint) if they
cannot both occur at the same time:

A ∩ B = ∅ or P(A ∩ B) = 0

Collectively Exhaustive Events

At least one of the events must occur:

A ∪ B = S or P(A ∪ B) = 1

Questions: What about Mutually Exclusive and Collectively

Exhaustive events?

12/38
Concepts of Probability

Probability calculation
In case of equally likely outcomes, the probability of a particular
event is:
Number of elements in the event n(A)
P(A) = Number of elements in the sample space = n(S)

Example
Experiment: tossing a six-sided dice
I Sample space: S = {1, 2, 3, 4, 5, 6}
I Event: the number shown is odd: A = {1, 3, 5}
n(A) 3 1
I Probability: P(A) = n(S) = 6 = 2

13/38
Concepts of Probability
Probability calculation
In case of unequally likely outcomes, the probability of a particular
event is:
I Assign probabilities to each outcome and identify the event of
interest
I Add probabilities of relevant outcomes

Example
A bag contains 3 red balls, 2 blue balls, and 5 green balls. The
outcomes (drawing a red, blue, or green ball) are not equally likely
I Experiment: draw a ball from the bag
I Sample space: S={Red, Blue, Green}
I Event: Drawing a red or a green ball: A={Red, Green}
3 5 8
I Probability: P(A) = 10 + 10 = 10
14/38
Conditional Probability

Conditional Probability
I The probability of a particular event occurring, given that
another event has occurred
I If A and B are events in S, and P(B) > 0, then the
conditional probability of A given B, written as P(A|B), is
P(A and B) P(A∩B)
P(A|B) = P(B) = P(B)

15/38
Conditional Probability
Contingency Table
A table used to classify some observations according to two or
more identifiable categories or classes
Education Male Female Total
Primary 10 10 20
Junior 20 10 30
Senior 20 30 50
College 60 40 100
Total 110 90 200

110
I The probability of a male respondent (event A) is 200
100
I The probability of having college education (event B) is 200
I As a result, the probability of having college education, given
male respondent, is
60
P(A∩B) 6
P(B|A) = P(A) = 200
110 = 11
200
16/38
Conditional Probability

Marginal Probability
The probability of an event occurring unconditional on any other
events
K
P(A ∩ Bk )
P
P(A) =
i=1

I B1 , B2 , . . . , BK are mutually exclusive and collectively

exhaustive events

17/38
Conditional Probability
Education Male Female Total
Primary 10 10 20
Junior 20 10 30
Senior 20 30 50
College 60 40 100
Total 110 90 200

Example
I The probability of a male respondent is unconditional on
education

P(Male) =P(Male and Primary) + P(Male and Junior)+

P(Male and Senior) + P(Male and College)
10 20 20 60 110
= + + + =
200 200 200 200 200
18/38
Independence

Independence
I The occurrence of one event has no effect on the probability
of the occurrence of another event
I Two events, A and B, are statistically independent if
P(A ∩ B) = P(A)P(B)

19/38
Independence

A ball is drawn twice, with replacement, from a bag containing 3

red balls and 7 yellow balls. What is the probability of selecting
two red balls?
I The probability of getting a red ball for the first time is:
3
P(A) = 10
I The probability of getting a red ball for the second time is:
3
P(B) = 10
I The probability of two red balls being selected is:
3 3 9
P(A ∩ B) = P(A)P(B) = 10 × 10 = 100
I Events A and B are independent

20/38
Rules of Addition
Joint Probability
A probability that measures the likelihood two or more events will
happen concurrently

General rule of Addition

P(A ∪ B) = P(A or B) = P(A) + P(B) − P(A ∩ B)

A∩B A∪B

A B A B

21/38
Rules of Addition

Example
What is the probability that a card chosen at random from a
standard deck of cards will be either a king or a heart?
4
I The probability of getting a king: P(A) = 52
I The probability of getting a heart: P(B) = 13
52
1
I The probability of getting a king of hearts: P(A ∩ B) = 52
I The probability of either a king or a heart:
4 13 1 16 4
P(A∪B) = P(A)+P(B)−P(A∩B) = 52 + 52 − 52 = 52 = 13

22/38
Rules of Addition

Mutually Exclusive Events

Two or more events are mutually exclusive (or disjoint) if they
cannot both occur at the same time:

P(A ∩ B) = 0

Special Rule of Addition (for mutually exclusive events)

P(A ∪ B) = P(A) + P(B) − P(A ∩ B) = P(A) + P(B)

23/38
Rules of Addition

Example
What is the probability that a card chosen at random from a
standard deck of cards will be either a king or a queen?
4
I The probability of getting a king: P(A) = 52
4
I The probability of getting a queen: P(B) = 52
I The probability of getting a king or queen:
4 4 8 2
P(A ∪ B) = P(A) + P(B) = 52 + 52 = 52 = 13

24/38
Rules of Addition

Complement Rule

P(A) = 1 − P(Ac )

Ac

25/38
Rules of Multiplication

General Rule of Multiplication

P(A ∩ B) = P(A and B) = P(A) × P(B|A)

Example
A ball is drawn twice, without replacement, from a bag containing
3 red balls and 7 yellow balls. What is the probability of selecting
two red balls?
I The event (A) that the first ball being selected is red:
3
P(A) = 10
I The event (B) that the second ball being selected is also red:
P(B|A) = 29
I The probability of two red balls being selected is:
3 2 6 1
P(A ∩ B) = P(A) × P(B|A) = 10 × 9 = 90 = 15

26/38
Rules of Multiplication

Special Rule of Multiplication (for independent events)

P(A ∩ B) = P(A)P(B)

See example on Independence

Test for Independence

I Two events, A and B, are statistically independent if
P(A ∩ B) = P(A)P(B)
I Calculate P(A ∩ B), P(A), and (B), and check whether
P(A ∩ B) = P(A)P(B) holds

27/38
Rules of Multiplication

Example
Different courses may be related, meaning students who take one
class might perform better in another. Assess whether there is a
dependency between performance in Statistics and Microeconomics

I The passing rate of Statistics is 90% last year

I The passing rate of Microeconomics is 80% last year
I 72% of the students passed both Statistics and
Microeconomics last year
I P(A) × P(B) = 0.9 × 0.8 = P(A ∩ B) = 0.72
I Therefore, Statistics and Microeconomics are independent to
each other

28/38
Rules of Multiplication
The Relationship between Mutually Exclusive Events and
Independent Events
I Mutually Exclusive Events
I P(A ∩ B) = 0
I P(A ∪ B) = P(A) + P(B) − P(A ∩ B) = P(A) + P(B)
I Independent Events
I P(A ∩ B) = P(A)P(B)
I P(A∪B) = P(A)+P(B)−P(A∩B) = P(A)+P(B)−P(A)P(B)
I The Relationship
I Mutually Exclusive events are not independent
I Independent events cannot be mutually exclusive
I Note
I A and B are Mutually Exclusive and Independent if and only if
P(A) = 0 or P(B) = 0
I Typically we are more interested in events with positive
probability as they are more meaningful
29/38
Bayes’ Theorem
Bayes’ Theorem
Bayes’ Theorem is used to update the probability of an event
based on new information that might be related to the event
P(Ai )P(B|Ai )
P(Ai |B) = P
n
P(Ai )P(B|Ai )
i=1

I A1 , A2 , . . . , and An are mutually exclusive and collectively

exhaustive events
I B is an event of interest that might affect P(Ai )
I For example, initially we know the probability of passing
Statistics. Later we learn that having a textbook is related to
a student’s grade. Then we can use this information to more
accurately calculate the the probability of passing Statistics
than can be done without knowledge of whether the student
has a textbook
30/38
Bayes’ Theorem

Prior Probability
The initial probability based on the present level of information

P(Ai )

Posterior Probability
A revised probability based on additional information

P(Ai |B)

31/38
Bayes’ Theorem
Derivation
I A1 , A2 , . . . , and An are mutually exclusive and collectively
exhaustive events
I B is an event of interest that might affect P(Ai )
I Conditional Probabilities:
P(Ai ∩B)
I P(Ai |B) = P(B)
P(B∩Ai )
I P(B|Ai ) = P(Ai )
I It follows that:
P(Ai ∩ B) = P(B)P(Ai |B) = P(Ai )P(B|Ai )

P(Ai )P(B|Ai )
P(Ai |B) =
P(B)
P(Ai )P(B|Ai )
=P
n
P(Ai )P(B|Ai )
i=1

32/38
Bayes’ Theorem
Example
Suppose that 5% of the population are drug users. A drug test will
produce 90% true positive results for drug users and 15% false
positive results for non-drug users. If a randomly selected
individual tests positive, what is the probability he is a drug user?
I Let A1 be (drug) users, and A2 be non-users
I B is being tested positive
I By using the Bayes’ theorem:

P(A1 )P(B|A1 )
P(A1 |B) =
P(A1 )P(B|A1 ) + P(A2 )P(B|A2 )
P(users)P(positive|users)
=
P(users)P(positive|users) + P(nonusers)P(positive|nonusers)
0.05 × 0.90
=
0.05 × 0.90 + 0.95 × 0.15
=24%
33/38
Bayes’ Theorem

Example
Suppose that 5% of the population are drug users. A drug test will
produce 90% true positive results for drug users and 15% false
positive results for non-drug users. If a randomly selected
individual tests positive, what is the probability he is a drug user?

Prior Conditional Joint Posterior

Event
Probability Probability Probability Probability
Ai P(Ai ) P(B|Ai ) P(Ai ∩ B) P(Ai |B)
A1 0.05 0.90 0.0450 0.0450/0.1875=0.24
A2 0.95 0.15 0.1425 0.1425/0.1875=0.76

What is the probability of a randomly selected individual being

tested positive, i.e., P(B)?

34/38
Bayes’ Theorem

Exercise 1
Suppose 55% of AGEC students are advised by Dr. A and 45% by
Dr. B. On average, Dr. A holds advisory gatherings with a 90%
probability, while Dr. B holds them with a 10% probability. If a
randomly selected AGEC student had an advisory gathering with
their advisor, what is the probability that their advisor is Dr. B?
I Define events
I A1 =
I A2 =
I B=

35/38
Bayes’ Theorem

Exercise 1
Suppose 55% of AGEC students are advised by Dr. A and 45% by
Dr. B. On average, Dr. A holds advisory gatherings with a 90%
probability, while Dr. B holds them with a 10% probability. If a
randomly selected AGEC student had an advisory gathering with
their advisor, what is the probability that their advisor is Dr. B?

Prior Conditional Joint Posterior

Event
Probability Probability Probability Probability
Ai P(Ai ) P(B|Ai ) P(Ai ∩ B) P(Ai |B)
A1
A2

36/38
Bayes’ Theorem

Exercise 2
Suppose, on average, 70% of AGEC students take Statistics at
AGEC, 20% at Accounting, and 10% in Economics, with historical
failure rates of 5%, 10%, and 20%, respectively. If a randomly
selected AGEC student fails Statistics, what is the probability that
they took the class at AGEC?
I Define events
I A1 =
I A2 =
I A3 =
I B=

37/38
Bayes’ Theorem

Exercise 2
Suppose, on average, 70% of AGEC students take Statistics at
AGEC, 20% at Accounting, and 10% in Economics, with historical
failure rates of 5%, 10%, and 20%, respectively. If a randomly
selected AGEC student fails Statistics, what is the probability that
they took the class at AGEC?

Prior Conditional Joint Posterior

Event
Probability Probability Probability Probability
Ai P(Ai ) P(B|Ai ) P(Ai ∩ B) P(Ai |B)
A1
A2
A3

38/38