ABSTRACT
The Bayesian approach to statistical inference is quite different from the classical approach, though they both depend essentially upon the likelihood. As stated in Section 1.4, in the classical approach the model parameters are regarded as fixed quantities that need to be estimated, and this is typically done by finding the values of the parameters that maximise the likelihood, considered as a function of the parameters. As we have seen, this usually reduces to a problem in numerical optimisation, often in a high dimensional space. In the Bayesian approach, parameters are seen as variable and possessing a distribution, on the same footing as the data. Before data are collected, it is the prior distribution that describes the variation in the parameters. After the collection of data the prior is replaced by a posterior distribution. No optimisation is involved. As we shall see, the approach follows from a simple application of Bayes’ Theorem, however the problem with the use of Bayes’ Theorem is that it provides a posterior distribution for all of the model parameters jointly, when we might be especially interested in just a subset of them. For example, if we are interested in only a particular parameter, we want to estimate the marginal posterior distribution for that parameter alone. In order to obtain the marginal posterior distribution of the parameters of interest, we have to do what we always do when seeking a marginal distribution, which is integrate the joint posterior distribution. It is here that the difficulties typically arise when a Bayesian analysis is undertaken. The optimisation of classical analysis has been replaced by integration for the Bayesian approach. The modern approach to Bayesian analysis is not to try to integrate the posterior joint distribution analytically, but instead to employ special simulation procedures which result in samples from the posterior distribution. Having simulated values from the posterior joint distribution means that one then naturally has simulated values from the posterior marginal distribution of the parameters of interest. This approach is called Markov chain Monte Carlo (MCMC), and it will be described in detail in Chapter 5.
