ABSTRACT
Consider the simplest case where time is discrete, and both the state-space X and the control set U are finite. Suppose that the system is time invariant and evolves probabilistically as a controlled Markov chain, i.e.,
Prob[x(t + 1) = j|x(t) = i, u(t) = u] = pij(u). (61.7) The elements of the matrix P(u) = [pij(u)] are the transition probabilities.
