ABSTRACT
A Markov process is a mathematical model, based on principles developed by the Russian probability theorist A. A. Markov, which allows systems engineers and analysts to describe and predict the behavior of such systems. Markov analysis has been found to be useful in areas as disparate as population dynamics, inventory management, equipment maintenance and replacement problems, market share analysis, and economic trend analysis. This chapter begins with some preliminary definitions, and then investigates the types of analysis that can be performed. It should be noted that the validity of any study using the tools of Markov analysis hinges on the extent to which the Markov and stationarity assumptions are met by the actual system under investigation. The calculations required for finding steady-state probabilities and expected first passage times are just the standard procedures for solving systems of linear equations. In the Markov decision process, not every decision is allowable in every state or with every possible state transition.
