Post on 03-Jul-2020
c©Stanley Chan 2019. All Rights Reserved.
ECE 302: Chapter 02 Probability Model
Fall 2019
Prof Stanley Chan
School of Electrical and Computer EngineeringPurdue University
1 / 52
c©Stanley Chan 2019. All Rights Reserved.
1. Set Theory
2 / 52
c©Stanley Chan 2019. All Rights Reserved.
Set
Definition
A set is a collection of objects. We denote A = ξ1, ξ2, . . . , ξn as a set,and ξi be the i-th element in the set.
Notation:
ξ ∈ A: An object ξ is in set A.
ξ 6∈ A: An object ξ is not in set A.
Finite, Countable, Uncountable.
Finite: A = 0, 1Countable: A = 2, 4, 6, 8, . . .Uncountable: A = x | 0 < x < 1
Open and Closed Intervals.
(a, b) = x | a < x < b[a, b] = x | a ≤ x ≤ b
3 / 52
c©Stanley Chan 2019. All Rights Reserved.
Subset
Definition
A subset of A is sub-collection of objects in A. That is, B ⊆ A if for anyξ ∈ B, this ξ is also in A.
Proper subset: B ⊂ A.
Example. If A = 1, 2, 3, 4, 5, 6, then B = 1, 2, is a proper subset of A.Example. If A = 1, 2, 3, then the set B = 1, 2, 3 is an impropersubset of A.
Theorem
If A ⊆ B and B ⊆ A, then A = B.
Proof. Suppose A ⊂ B (which means A 6= B. You may also considerB ⊂ A). Then there exist x ∈ B but 6∈ A. But B ⊆ A requires all x ∈ Balso ∈ A. So we reach a contradiction. The only way to resolve thecontradiction is to make A = B.
4 / 52
c©Stanley Chan 2019. All Rights Reserved.
Empty and Universal Set
Definition (Empty Set)
A set is empty if it contains no element. We denote an empty set as ∅.
An empty set is a subset of any set.
Definition
The universal set is the set containing all elements. We denote auniversal set as Ω.
Any set is a subset of Ω, including Ω itself.
5 / 52
c©Stanley Chan 2019. All Rights Reserved.
Finite Union
Definition (Finite Union)
The finite union of two sets A and B contains all elements in A or in B.That is,
A ∪ B = ξ | ξ ∈ A or ξ ∈ B.
Example. If A = 1, 2, 3, 4, B = 1, 5, 6, then A ∪ B =
Example. If A = t | 3 < t ≤ 4, B = t | t ≥ 3.5, then A ∪ B =
6 / 52
c©Stanley Chan 2019. All Rights Reserved.
Infinite Union
Definition (Infinite Union)
The infinite union of A1,A2, . . . ,An is denoted as
Adef=
∞⋃n=1
An.
It holds that x ∈ A if x is in at least one of A1,A2, . . . ,An.
Example. If An =[0, 1− 1
n
], then
(a) A = [0, 1](b) A = [0, 1)
7 / 52
c©Stanley Chan 2019. All Rights Reserved.
Finite Intersection
Definition
The finite intersection of two sets A and B contains all elements in Aand in B. That is,
A ∩ B = ξ | ξ ∈ A and ξ ∈ B.
Example. If A = 1, 2, 3, 4, B = 1, 5, 6, then A ∩ B =
Example. If A = t | 3 < t ≤ 4, B = t | t ≥ 3.5, then A ∩ B =
8 / 52
c©Stanley Chan 2019. All Rights Reserved.
Infinite Intersection
Definition
Define the infinite intersection of A1,A2, . . . ,An as
A =∞⋂n=1
An
Then, x ∈ A if x is in all of A1,A2, . . . ,An.
Example. If An = [0, 1 + 1n ), then
(a) A = [0, 1](b) A = [0, 1)
9 / 52
c©Stanley Chan 2019. All Rights Reserved.
Complement
Definition
The complement of a set A is the set containing all elements in Ω but notin A. That is
Ac = ξ | ξ ∈ Ω and ξ 6∈ A.
Example. Let A = even integers, Ω = integers, then Ac =
10 / 52
c©Stanley Chan 2019. All Rights Reserved.
Difference
Definition
The difference A\B is the set containing all elements in A but not in B.
A\B = ξ | ξ ∈ A and ξ 6∈ B.
11 / 52
c©Stanley Chan 2019. All Rights Reserved.
Disjoint
Definition
Two sets A and B are disjoint if A ∩ B = ∅. For a collection of setsA1,A2, . . . ,An, we say that the collection is disjoint if Ai ∩ Aj = ∅.
12 / 52
c©Stanley Chan 2019. All Rights Reserved.
Partition
Definition
A collection of sets A1, . . . ,An is a partition to the universal set Ω if itsatisfies the following conditions:
(non-overlap) A1, . . . ,An is disjoint;
(decompose) A1 ∪ A2 ∪ . . . ∪ An = Ω.
13 / 52
c©Stanley Chan 2019. All Rights Reserved.
Set Operations
There are four set operations:
Commutative (Order does not matter).
A ∩ B = B ∩ A
A ∪ B = B ∪ A
Associative (How to do multiple union and intersection)
A ∪ (B ∪ C ) = (A ∪ B) ∪ C
A ∩ (B ∩ C ) = (A ∩ B) ∩ C
14 / 52
c©Stanley Chan 2019. All Rights Reserved.
Set Operations
Distributive (How to mix union and intersection)
A ∩ (B ∪ C ) = (A ∩ B) ∪ (A ∩ C )
A ∪ (B ∩ C ) = (A ∪ B) ∩ (A ∪ C )
De Morgan’s Law (How to complement over intersection and union)
(A ∩ B)c = Ac ∪ Bc
(A ∪ B)c = Ac ∩ Bc
15 / 52
c©Stanley Chan 2019. All Rights Reserved.
2. Probability Model
16 / 52
c©Stanley Chan 2019. All Rights Reserved.
What is Probability?
It is a number.
Always between 0 and 1.
Always the probability of an event.
Example. The probability of getting a Head when tossing a coin:
P(“H”) =
17 / 52
c©Stanley Chan 2019. All Rights Reserved.
Three Elements of a Probability Model
1 Sample Space
2 Event
3 Probability Law
18 / 52
c©Stanley Chan 2019. All Rights Reserved.
Sample Space
Definition (Sample Space)
A sample space Ω is the collection of all possible outcomes.
We denote ω as an element in Ω.
Example.
Coin flip:Ω =
Throw a dice:Ω =
Waiting time for a bus in West Lafayette:Ω =
19 / 52
c©Stanley Chan 2019. All Rights Reserved.
Event
Definition (Event)
An event F is a subset in the sample space Ω.
Outcome VS Event:
Example. Throw a dice. Let Ω = 1, 2, 3, 4, 5, 6.
F1 = even numbers = 2, 4, 6.F2 = less than 3 = 1, 2.
Example. Wait a bus. Let Ω = 0 ≤ t ≤ 30.F1 = 0 ≤ t < 10F2 = 0 ≤ t < 5 ∪ 20 < t ≤ 30.
20 / 52
c©Stanley Chan 2019. All Rights Reserved.
Event Space
Definition (Event Space)
The collection of all possible events is called the
Event Space or σ-field
denoted as F . F satisfies the following two properties:
If F ∈ F , then F c ∈ F
If F1,F2, . . . ∈ F , then Fi ∩ Fj ∈ F and Fi ∪ Fj ∈ F .
Example. Ω = H,T, the event space is ∅,H,T ,Ω.
21 / 52
c©Stanley Chan 2019. All Rights Reserved.
Probability Law
Definition
A probability law is a function P : F → [0, 1] that maps an event A to areal number in [0, 1]. The function must satisfy three axioms known asProbability Axioms.
I. Non-negativity:
II. Normalization:
22 / 52
c©Stanley Chan 2019. All Rights Reserved.
Probability Law
III. Additivity:
For any disjoint subsets A1,A2, . . ., it holds that
P
[ ∞⋃n=1
An
]=
∞∑n=1
P[An].
23 / 52
c©Stanley Chan 2019. All Rights Reserved.
Properties of Probability
1 P[Ac ] = 1− P[A].
2 For any A ⊆ Ω, P[A] ≤ 1.
3 P[∅] = 0.
24 / 52
c©Stanley Chan 2019. All Rights Reserved.
Properties of Probability
4 For any A and B,
P[A ∪ B] = P[A] + P[B]− P[A ∩ B].
25 / 52
c©Stanley Chan 2019. All Rights Reserved.
Properties of Probability
5 (Union Bound) For any A and B,
P[A ∪ B] ≤ P[A] + P[B].
26 / 52
c©Stanley Chan 2019. All Rights Reserved.
Properties of Probability
6 If A ⊆ B, then P[A] ≤ P[B]
Example. A = t ≤ 5, and B = t ≤ 10, then P[A] ≤ P[B].
27 / 52
c©Stanley Chan 2019. All Rights Reserved.
Example
Let the events A and B have P[A] = x , P[B] = y and P[A ∪ B] = z . Findthe following probabilities.
(a) P[A ∩ B]
(b) P[Ac ∩ Bc ]
28 / 52
c©Stanley Chan 2019. All Rights Reserved.
Example
(c) P[Ac ∪ Bc ]
(d) P[A ∩ Bc ]
29 / 52
c©Stanley Chan 2019. All Rights Reserved.
Probability as a Measure
P[·] is a measure of the size of the event.
Ω = 1, 2, 3, 4, 5, 6The measure P[·] is acounter
Counts the number of events
Ω = [0, 1]× [0, 1]
The measure P[·] is anintegration
Integrates the area of events
30 / 52
c©Stanley Chan 2019. All Rights Reserved.
Probability as a Measure
P[·] is a measure of the size of the event.
Ω = 1, 2, 3, 4, 5, 6E = 1, P[E ] = 1/6
E = 1, 3, P[E ] = 2/6
P[E1] ≤ P[E2] if E1 ⊆ E2.
Ω = [0, 1]× [0, 1]
E = shaded region, P[E ] = area.
E = (x0, y0), P[E ] = 0.
P[E ] can be 0 even if E 6= ∅.
31 / 52
c©Stanley Chan 2019. All Rights Reserved.
Measure Zero
Definition
Let Ω be the sample space. A set A ∈ Ω is said to have measure zero iffor any given ε > 0,
There exists a countable number of subsets An such thatA ⊆ ∪∞n=1An, and∑∞
n=1 P[An] < ε.
32 / 52
c©Stanley Chan 2019. All Rights Reserved.
Measure Zero Example
Example. Let Ω = [0, 1]. Then the set 0.5 ⊂ Ω has measure zero.
Fix any ε.
Create a very small interval around 0.5.
Shrink the radius of the interval to make sure it is less than ε.
Facts.
Discrete space: No issue about measure zero.
Example. 1 in 1, . . . , 6 has non-zero probability.
Continuous space: Isolated points have measure zero.
Example. 1 in [1, 6] has zero probability.
33 / 52
c©Stanley Chan 2019. All Rights Reserved.
Almost Surely
Definition
An event A ∈ R is said to hold almost surely (a.s.) if
P[A] = 1,
except for all measure-zero sets in R.
Example. Let Ω = [0, 1]. Then
P[(0, 1)] = 1 (almost surely)
because the points 0 and 1 have measure zero in Ω.
Note: In this course we will skip “a.s.” if the context is clear.
34 / 52
c©Stanley Chan 2019. All Rights Reserved.
3. Conditional Probability
35 / 52
c©Stanley Chan 2019. All Rights Reserved.
Definition (Conditional Probability)
Assume P[B] 6= 0. The conditional probability of A given B is
Difference:
P[A |B] =P[A ∩ B]
P[B]and P[A ∩ B] =
P[A ∩ B]
P[Ω].
36 / 52
c©Stanley Chan 2019. All Rights Reserved.
Understanding Conditional Probability
https://ocw.mit.edu/resources/
res-6-012-introduction-to-probability-spring-2018/part-i-the-fundamentals/
37 / 52
c©Stanley Chan 2019. All Rights Reserved.
Example
https://ocw.mit.edu/resources/
res-6-012-introduction-to-probability-spring-2018/part-i-the-fundamentals/
38 / 52
c©Stanley Chan 2019. All Rights Reserved.
Example
Conditional probability can appear in daily life:
A = You are admitted to Harvard
B = Your SAT score is 2200 or above
The probabilities are:
P[A] very small (4.5% in 2019 1)
P[B] hard, but not too bad
P[A|B] may not be very high, because a lot of applicants areoutstanding
P[B|A] probably higher, because you will not be reviewed if your SATis low
And do not forget, the 4.5% is a conditional probability conditioned onthose who applied only.
1https://www.thecrimson.com/article/2019/3/29/2023-admit-numbers/39 / 52
c©Stanley Chan 2019. All Rights Reserved.
Bayes Theorem
Theorem (Bayes Theorem)
For any two events A and B such that P[A] > 0 and P[B] > 0, it holdsthat
P[A |B] =P[B |A]P[A]
P[B].
40 / 52
c©Stanley Chan 2019. All Rights Reserved.
Law of Total Probability
Theorem (Law of Total Probability)
Let A1,A2, . . . ,An be a partition of Ω, i.e., A1, . . . ,An are disjoint andΩ = A1 ∪ A2 ∪ . . . ∪ An. Then, for any B ⊆ Ω,
P[B] =n∑
i=1
P[B |Ai ] P[Ai ].
41 / 52
c©Stanley Chan 2019. All Rights Reserved.
Law of Total Probability
Figure: Law of total probability decomposes the probability P[B] into multipleconditional probabilities P[B |Ai ]. The probability of obtaining each P[B |Ai ] isP[Ai ].
42 / 52
c©Stanley Chan 2019. All Rights Reserved.
Example 1: Tennis Tournament
Consider a tennis tournament. Your probability of winning the game is
0.3 against1
2of the players (Event A).
0.4 against1
4of the players (Event B).
0.5 against1
4of the players (Event C).
What is the probability of winning the game?
43 / 52
c©Stanley Chan 2019. All Rights Reserved.
Example 2: Communication Channel
Consider a communication channel shown below. The probability ofsending a 1 is p and the probability of sending a 0 is 1− p. Given that 1 issent, the probability of receiving 1 is 1− η. Given that 0 is sent, theprobability of receiving 0 is 1− ε.
Find P[Receive 1]
Find P[Send 1]
Find P[Receive 1 | Send 1]
Find P[Send 1 | Receive 1]
44 / 52
c©Stanley Chan 2019. All Rights Reserved.
Properties of Conditional Probability
Theorem
Let P[B] > 0. The conditional probability P[A |B] satisfies Axiom I toAxiom III.
Proof.
Axiom I:
Axiom II:
Axiom III:
45 / 52
c©Stanley Chan 2019. All Rights Reserved.
Independence
Definition
Two events A and B are statistically independent if
Disjoint VS Independent.
46 / 52
c©Stanley Chan 2019. All Rights Reserved.
Examples of Independence
Example 1. Throw a dice twice. Let
A = 1st dice is 3 and B = 2nd dice is 4.Are A and B independent?
47 / 52
c©Stanley Chan 2019. All Rights Reserved.
Examples of Independence
Example 2. Throw a dice twice. Let
A = 1st dice is 1 and B = sum is 7.
Are A and B independent?
Think about P[A|B].
If you know the sum is 7, then the pair has to be (1,6), (2,5), (3,4),(4,3), (5,2), (6,1).
The chance of getting first dice = 1 is still 1/6. It has been not beenchanged by B.
So independent.
How about you change B = sum is 8?
48 / 52
c©Stanley Chan 2019. All Rights Reserved.
Examples of Independence
Example 3. Throw a dice twice. Let
A = 1st dice is 2 and B = sum is 8.
Are A and B independent?
Think about P[A|B].
If you know the sum is 8, then the pair has to be (2,6), (3,5), (4,4),(5,3), (6,2).
The chance of getting first dice = 2 is no longer 1/6. It has beenchanged by B.
So dependent.
49 / 52
c©Stanley Chan 2019. All Rights Reserved.
Examples of Independence
Example 4. Throw a dice twice. Let
A = max is 2 and B = min is 2.Are A and B independent?
50 / 52
c©Stanley Chan 2019. All Rights Reserved.
Independence Via Conditional Probability
Recall that P[A |B] = P[A∩B]P[B] .
If A and B are independent, then P[A ∩ B] = P[A]P[B]
Therefore,
P[A |B] =P[A ∩ B]
P[B]=
P[A] P[B]
P[B]= P[A].
Interpretation.
51 / 52
c©Stanley Chan 2019. All Rights Reserved.
Prisoner’s Dilemma
Three Prisoners: A, B, C. The King decides to release 2 and kill 1.You were A.Your chance of release is 2/3.Suppose you know the guard well. You can ask him about which of Bor C will be released.But if you find out B (or C) is released, your chance becomes 1/2.How come!!
52 / 52