ABSTRACT
When asked to write a survey chapter on information theory and neural nets, I was both intrigued and concerned. Intrigued, because I was hoping that information theory might provide some guidance to the understanding of these tricky but apparently powerful modeling devices and possibly to suggest even new ideas for their design. Concerned, because I had, and still have, only a superficial knowledge of neural nets, and I had found the reading of their vast literature tough going, which the mixture of well-known principles with ad hoc techniques couched in an applications-oriented language, made no easier. Although information theoretic notions appear quite frequently in the neural net literature, there are not many major areas of contact between the two disciplines. The intersection is not empty, however, and I have selected three main themes where information theory has already made an impact or, as I hope, it will prove beneficial to the neural network community.
