ABSTRACT
This chapter introduces the basic Bayesian theory. For an arbitrary triple of a true distribution, a statistical model, and a prior, the behaviors of the free energy or the minus log marginal likelihood, the generalization loss, cross validation loss, training loss, and WAIC are derived by the procedure. The chapter discusses the formal relation between a true distribution and a statistical model. It provides definitions of Bayesian observables and their normalized ones. The chapter examines the cumulant generating function of the Bayesian prediction. It explores the basic theory of Bayesian statistics which is proved by using the cumulant generating function. The chapter shows the recipe for the Bayesian theory construction and its application. In order to study the asymptotic behaviors of the generalization loss, the cross validation loss, the training loss, and widely applicable information criterion (WAIC), the cumulant generating functions are useful.
