ABSTRACT
To this point we have thoroughly discussed differential entropy and some of its properties. In this chapter we aim to provide the reader with a few applications of differential entropy to problems of interest in physics and engineering. To this point we have focused on the differential entropy of a single random variable or on the joint entropy among two random variables X, Y . In practice, however, we are typically dealing with temporal sequences of observations x(t1), x(t2), · · · , x(tN ), y(t1), y(t2), · · · , y(tN ). From Chapter 1 we know that such sequences are modeled as random processes, that is, with the vector of random variables X, Y. The entropies associated with these random processes are therefore given by multi-dimensional integrals over the joint PDFs, e.g., hX =
∫ RN pX(x) log2 (pX(x)) dx. Computing the entropy for a random
process is therefore analytically intractable for nearly all joint distributions. An exception is the joint Gaussian distribution for which we will provide a derivation of diffential entropy shortly.
