Scribd
Upload
78 views

STA 348 Introduction To Stochastic Processes

This document discusses continuous-time Markov chains (CMC), which are stochastic processes where the time between state transitions is modeled by a continuous random variable rather than discrete time steps. It provides examples of modeling different processes as CMCs, such as a population model where individuals arrive and depart according to exponential rates. Key aspects of CMCs discussed include the Markov property for continuous time, specifying CMCs using transition rates between states, and birth-death processes as a special case of CMCs.

Uploaded by

Mercy
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
78 views

STA 348 Introduction To Stochastic Processes

This document discusses continuous-time Markov chains (CMC), which are stochastic processes where the time between state transitions is modeled by a continuous random variable rather than discrete time steps. It provides examples of modeling different processes as CMCs, such as a population model where individuals arrive and depart according to exponential rates. Key aspects of CMCs discussed include the Markov property for continuous time, specifying CMCs using transition rates between states, and birth-death processes as a special case of CMCs.

Uploaded by

Mercy
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 14

STA 348

Introduction to
Stochastic Processes
Lecture 17

1
Example
 Suppose customers arrive at store according to
PP with rate λ, and the time each customer
spends in store follows some distribution with cdf
G(x). Let X(t) be number of customers in store at
time t; what is the distribution of X(t) & N(t)−X(t)?

2
Example
 Let N(t) be PP counting # people infected with
some virus, whose symptoms appear after an
Exp(µ=.1) amount of time. If after t=16 years
there are 220,000 infections with symptoms, try
to estimate the # of symptomless infections.

3
4
5
Continuous Markov Chains
 A continuous Markov Chain (CMC) {X(t),t≥0}
is like MC where the time between transitions
is a continuous RV, and not a fixed time step
 State-space of X(t) is non-negative integers  0
 Index set of X(t) is non-negative reals  0
 The process satisfies the Markov property:
P  X (t  s )  j | X ( s )  i,{ X (u )  ku , 0  u  s} 
 P  X (t  s )  j | X ( s)  i 

6
Example
 Sample path of CMC {X(t),t≥0}:

time between
transitions X (t )

4
3
2
1
t (time)
0
7
Stationary CMC
 We will only look at stationary CMC’s
 P  X (t  s )  j | X ( s )  i   P  X (t )  j | X (0)  i 
 I.e. probability of going from state i to state j in
time t is independent of when we start (time s)
 What does that tell us about the time
between transitions?
 The time between transitions follows an
exponential distribution
 The rate of the exponential depends on the state
from which we make the transition
8
Stationary CMC
 Consider stationary CMC {X(t),t≥0}, and
condition on X(s)=i. Let Ti be the transition
time from i (i.e. time it takes to leave state i).
 Show that Ti~Exponential

9
Alternative Specification of
CMC’s
 A continuous-time, discrete state-space SP
{X(t),t≥0} is a CMC if:
 The amount of time it spends in state i before it
transitions to another state is exponentially
distributed with some rate vi (indep. of other times)
 When the process leaves state i, it enters state j
with some probability Pij, where Pii  0 &  j Pij  1
 Similar to MC, but with time between
transition to different state being Exponential
10
Example
 Describe a PP with rate λ as a CMC (i.e. find
vi and Pij, for any i,j=0,1,2,…).

11
Example
 Shoe shine shop has two chairs, one for brushing (1) and
another for polishing (2). Customers arrive according to
PP with rate λ, and enter only if both chairs are empty.
Shoe-shiner takes Exp(µ1) time for brushing and Epx(µ2)
time for polishing. Describe the process with a CMC.

12
Birth & Death Process
 A Birth & Death Process is a CMC where for
each transition there are only 2 options
 (birth): go from state i to state i+1
 From 0, there can only be a birth (i.e. go to 1)
 (death): go from state i to state i−1
 Birth & Death processes used to model # of
individuals in a system, where individuals
arrive at some (state-dependent) exponential
rate λi, and exit at some exponential rate µi
 E.g. PP has birth rate λ and death rate 0 13
Birth & Death Process
 Describe Birth & Death process with birth rates
λi & death rates µi (µ0=0), as CMC

14

You might also like