A Dynamic Mobility Histogram Construction Method Based on Markov Chains
CS433 Modeling and Simulation Lecture 06 – Part 03 Discrete Markov Chains
description
Transcript of CS433 Modeling and Simulation Lecture 06 – Part 03 Discrete Markov Chains
CS433Modeling and Simulation
Lecture 06 – Part 03 Discrete Markov
Chains
Dr. Anis Koubâa12 Apr 2009
Al-Imam Mohammad Ibn Saud UniversityAl-Imam Mohammad Ibn Saud University
2
Classification of States: 1 A path is a sequence of states, where each transition has a
positive probability of occurring. State j is reachable (or accessible) (يمكن الوصول إليه) from state i
(ij) if there is a path from i to j –equivalently Pij (n) > 0 for some n≥0,
i.e. the probability to go from i to j in n steps is greater than zero. States i and j communicate (ij) (يتصل) if i is reachable from j and
j is reachable from i.
(Note: a state i always communicates with itself)
A set of states C is a communicating class if every pair of states in C communicates with each other, and no state in C communicates with any state not in C.
3
Classification of States: 1 A state i is said to be an absorbing state if pii = 1. A subset S of the state space X is a closed set if no
state outside of S is reachable from any state in S (like an absorbing state, but with multiple states), this means pij =
0 for every i S and j S
A closed set S of states is irreducible(غير قابل للتخفيض) if any state j S is reachable from every state i S.
A Markov chain is said to be irreducible if the state space X is irreducible.
4
Example Irreducible Markov Chain
0 1 2p01 p12
p00p10
p21
p22
p01 p12
p00p10
p14
p224
p23
p32
p33
0 1 2 3
Absorbing State
Closed irreducible set
Reducible Markov Chain
5
Classification of States: 2 State i is a transient state (حالة عابرة)if there exists a state j such that j is
reachable from i but i is not reachable from j. A state that is not transient is recurrent (حالة متكررة) . There are two types
of recurrent states:1. Positive recurrent: if the expected time to return to the state is finite.
2. Null recurrent (less common): if the expected time to return to the state is infinite (this requires an infinite number of states).
A state i is periodic with period k >1, if k is the smallest number such that all paths leading from state i back to state i have a multiple of k transitions.
A state is aperiodic if it has period k =1.
A state is ergodic if it is positive recurrent and aperiodic.
6
Classification of States: 2
Example from BookIntroduction to Probability: Lecture Notes
D. Bertsekas and J. Tistsiklis – Fall 200
7
Transient and Recurrent States
We define the hitting time Tij as the random variable that represents the time to go from state j to stat i, and is expressed as:
k is the number of transition in a path from i to j. Tij is the minimum number of transitions in a path from i to j.
We define the recurrence time Tii as the first time that the Markov Chain returns to state i.
The probability that the first recurrence to state i occurs at the nth-step is
Ti Time for first visit to i given X0 = i. The probability of recurrence to state i is
0min 0 : |ij kT k X j X i
( )1 1 0
0
Pr , ,..., |
Pr |
nii ii n n
i
f T n P X i X i X i X i
T n X i
0min 0 : |ii kT k X i X i
( )
1
Pr ni ii ii ii
n
f f T f
8
Transient and Recurrent States
The mean recurrence time is
A state is recurrent if fi=1
If Mi < then it is said Positive Recurrent If Mi = then it is said Null Recurrent
A state is transient if fi<1
If , then is the probability of never returning to state i.
0Pr Pr | 1i ii if T T X i
0Pr Pr | 1i ii if T T X i
( )0
0
| ni ii i ii
n
M E T E T X i n f
1if 1 Pri iif T
9
Transient and Recurrent States
We define Ni as the number of visits to state i given X0=i,
Theorem: If Ni is the number of visits to state i given X0=i,
then
Proof
( )0
0
1|1
ni ii
n i
E N X i Pf
0
1 if
0 if n
i n ni n
X iN I X i where I X i
X i
( )niiP Transition Probability from
state i to state i after n steps
10
Transient and Recurrent States
The probability of reaching state j for first time in n-steps
starting from X0 = i.
The probability of ever reaching j starting from state i is
( )1 1 0Pr , ,..., |n
ij ij n nf T n P X j X j X j X i
( )
1
Pr nij ij ij
n
f T f
11
Three Theorems
If a Markov Chain has finite state space, then: at least one of the states is recurrent.
If state i is recurrent and state j is reachable from state i then: state j is also recurrent.
If S is a finite closed irreducible set of states, then: every state in S is recurrent.
12
Positive and Null Recurrent States
Let Mi be the mean recurrence time of state i
A state is said to be positive recurrent if Mi<∞. If Mi=∞ then the state is said to be null-recurrent. Three Theorems
If state i is positive recurrent and state j is reachable from state i then, state j is also positive recurrent.
If S is a closed irreducible set of states, then every state in S is positive recurrent or, every state in S is null recurrent, or, every state in S is transient.
If S is a finite closed irreducible set of states, then every state in S is positive recurrent.
1
Pri ii iik
M E T k T k
13
Example
p01 p12
p00p10
p14
p224
p23
p32
p33
0 1 2 3
Recurrent State
Transient States Positive
Recurrent States
14
Periodic and Aperiodic States Suppose that the structure of the Markov Chain is
such that state i is visited after a number of steps that is an integer multiple of an integer d >1. Then the state is called periodic with period d.
If no such integer exists (i.e., d =1) then the state is called aperiodic.
Example1 0.5
0.5
0 1 21
Periodic State d = 2
0 1 00.5 0 0.50 1 0
P
15
Steady State Analysis Recall that the state probability, which is the probability
of finding the MC at state i after the kth step is given by:
Pri kX ik 0 1, ...k k k π An interesting question is what happens in the
“long run”, i.e., limi kk
Questions: Do these limits exists? If they exist, do they converge to a legitimate probability
distribution, i.e., How do we evaluate πj, for all j.
1i
This is referred to as steady state or equilibrium or stationary state probability
16
Steady State Analysis Recall the recursive probability
1k kπ π P If steady state exists, then π(k+1) π(k), and
therefore the steady state probabilities are given by the solution to the equations
If an Irreducible Markov Chain, then the presence of periodic states prevents the existence of a steady state probability
Example: periodic.m
π πP and 1ii
0 1 00.5 0 0.50 1 0
P 1 0 00 π
17
Steady State Analysis THEOREM: In an irreducible aperiodic Markov
chain consisting of positive recurrent states a unique stationary state probability vector π exists such that πj > 0 and
1limj jkj
kM
where Mj is the mean recurrence time of state j
The steady state vector π is determined by solving π πP and 1i
i
Ergodic Markov chain.
18
Discrete Birth-Death Example
1-p 1-p
pp
1-p
p0 1 i
p1 0
0 10 0
p pp p
p
P
Thus, to find the steady state vector π we need to solve π πP and 1i
i
19
Discrete Birth-Death Example
0 0 1p p In other words
1 1 , 1, 2,...1j j j p jp
1 01 p
p
Solving these equations we get2
2 0
1 pp
In general
0
1 j
j
pp
Summing all terms we get
0 00 0
1 11 1
i i
i i
p ppp
20
Discrete Birth-Death Example
Therefore, for all states j we get
0
1 i
i
pp
If p<1/2, then 0, for all j j
0
1 1j i
ji
p pp p
All states are transient
0
10
2 1
i
i
p pp p
If p>1/2, then 12 1 , for all
j
j
pp jpp
All states are positive recurrent
21
Discrete Birth-Death Example
If p=1/2, then
0
1 i
i
pp
0, for all j j
All states are null recurrent
22
Reducible Markov Chains
In steady state, we know that the Markov chain will eventually end in an irreducible set and the previous analysis still holds, or an absorbing state.
The only question that arises, in case there are two or more irreducible sets, is the probability it will end in each set
Transient Set T
Irreducible Set S1
Irreducible Set S2
23
Transient Set T
Reducible Markov Chains
Suppose we start from state i. Then, there are two ways to go to S. In one step or Go to r T after k steps, and then to S.
Define
Irreducible Set S
i
rs1
sn
0Pr | , 1, 2,...i kX S X i kS
24
Reducible Markov Chains
Next consider the general case for k=2,3,… 1 0Pr |X S X i ij
j S
p
1 1 1 0Pr , ..., |k k kX S X r T X r T X i
1 1 1 0
1 0
Pr , ...,| ,
Pr |k k kX S X r T X r T X i
X r T X i
i ij r irj S r T
p pS S
First consider the one-step transition
1 1 1Pr , ...,|k k k irX S X r T X r T p r irpS