5 Markov Chain and Hidden Markov Sequence : In this section, we provide basics for two very important random sequences related to the general Markov sequence discussed above. 3.5.1 Markov chain as discrete-state Markov sequence A Markov chain or discrete-state Markov sequence is a special case of a general Markov sequence. The state space of a Markov chain is of a discrete nature and is finite: which is called the transition matrix of the Markov chain. Given the transition probabilities of a Markov chain, the state-occupation probability can be easily computed. The computation is recursive according to

doi:10.1201/9781482276237-19

Chapter

5 Markov Chain and Hidden Markov Sequence

Chapter

In this section, we provide basics for two very important random sequences related to the general Markov sequence discussed above. 3.5.1 Markov chain as discrete-state Markov sequence A Markov chain or discrete-state Markov sequence is a special case of a general Markov sequence. The state space of a Markov chain is of a discrete nature and is finite: which is called the transition matrix of the Markov chain. Given the transition probabilities of a Markov chain, the state-occupation probability can be easily computed. The computation is recursive according to

ABSTRACT

The stationary distribution of a Markov chain plays an important role in a class of powerful statistical methods collectively named "Markov chain Monte Carlo (MCMC) methods. These methods are used to simulate (i.e., to sample or to draw) arbitrarily complex distributions, enabling one to carry out many difficult statistical inference and learning tasks which would otherwise be mathematically intractable. The theoretical foundation of the MCMC methods is the asymptotic convergence of a Markov chain to its stationary distribution, 7r(s(i)). That is, regardless of the initial distribution, the Markov chain is an asymptotically unbiased draw from r(s(i)). Therefore, in order to sample from an arbitrarily complex distribution, p(s), one can construct a Markov chain, by designing appropriate transition probabilities, aii, so that its stationary distribution is

r(s) = P(s). Three other interesting and useful properties of a Markov chain can be easily derived.

First, the state duration in a Markov chain is an exponential or geometric distribution: