ABSTRACT
The two central concepts in information theory are the information of an event and the entropy of a random variable.
13.1 Information The informationof a probabilistic eventE , denoted Inf(E ),measures the amount of information that we gain when we learn E , starting from scratch (i.e., from some presumed body of background knowledge). Inf(E ) is measured in number of bits. As in probabilistic notation, “E ,F” represents the event that both E and F are true; thus, Inf(E ,F ) is the information gained by learning both E and F . The conditional information Inf(E |F ) is the amount of information that we gain if we learn E after we have already learned F . Therefore, Inf(E |F ) = Inf(E ,F )− Inf(F ).
