ABSTRACT
This chapter introduces the concepts of the stochastic process and Markov chains. It focuses on the discrete-time Markov chain, DTMC, by discussing the respective Chapman-Kolmogorov equation and transient and steady-state solutions. The chapter examines the concepts of mean recurrence time, mean first passage time, holding time, and mean time to absorption. Markov chains have been applied in many areas of science and engineering. The complete probabilistic description of a stochastic process might be provided by the joint cumulative probability distribution function of its random variables. A process whose progress depends on the time at which it started its execution is said to be nonstationary. In Markov chains, state transitions depend on the current state and neither on the history that led to the current state nor on time already spent in the current state.
