ABSTRACT
This chapter introduces the Markov chains as simplest examples of discrete time stochastic processes. Markov chain models where introduced in the 1920s by A. A. Markov (Calculus of Probabilities,) 3rd ed., St. Petersburg, 1913. Informally a Markov chain is simply a sequence of random variables evolving with time. The random states are defined sequentially based on the current state and some additional random variables. The theory of Markov processes has led to rather intense activity in various scientific disciplines, providing natural probabilistic interpretations of various random evolution models arising in engineering, physics, biology and many other scientific disciplines. The elementary transitions of self-interacting processes depend on the history of the process, and more particularly on the occupation measure of the chain from the origin up to the present time. The resulting process can be interpreted as the motion of a single individual evolving with reinforced learning type strategies.
