ABSTRACT

Markov decision processes are extensively used in applications with sequential decision making. Yet there is no book or technical paper that explains how to write the code for a Markov Decision process. Therefore, this document presents a systematic way of writing MATLAB code for a Markov decision process (MDP). Guidelines are provided using a modular approach of writing the code that facilitates debugging. Value iteration algorithm has been used for calculation of optimal decision policy for the MDP. However general guidelines provided here are applicable to other methods of solving a Markov decision process e.g. policy iteration and linear programming. Writing the code for analysis of an MDP policy has also been discussed. Examples are provided throughout for better understanding of the methodology. Furthermore, MDP code for two case studies from international conference papers has been included for the purpose of demonstration.