ABSTRACT
In this chapter, our setting is we have a system that consists of a nite numberofstatess1, … , sn and a process that is in exactly one of these states at any giventime.Foradiscrete-timeMarkovchain,onecanvisualizetheprocess runningaccordingtoadigitalclockandtheprocessmakingthedecision ofhowtochangestatesateachchangeintheclock.Inmakingthechange ofstates,theprocessisgovernedbyasetofconditionalprobabilities,called transition probabilities. These are dened by
P ni j, ( ) = the probability that the process changes from state si to sj when the time changes from n to n + 1
P ni i, ( ) = the probability that the process remains in state si when the time changes from n to n + 1.
