Generative Models

We have previously seen some examples of generative models: the Naive Bayes classifier, for example, and the null category noise model. Generative models characterize the joint distribution p(x, y) of instances and labels, usually by separately characterizing the prior distribution of classes p(y) and the classspecific instance distribution or likelihood p(x|y). This is the source of the name: a generative model represents a hypothesis about how instances are generated for each class. For classification, the generative probability can be reversed using Bayes’ rule:

p(y|x) = p(y)p(x|y) p(x)

.