ABSTRACT
Introduction It is a great pleasure for me to write this contribution to honour Vela’s 60th birthday. Although our fields are somewhat different we both share the same interest in applying the ideas of the algorithmic theory of complexity to real world problems, he to economics and I to statistical inference. This has been a daunting task for both of us, for most people who are interested in finding solutions to their every day problems do not care whether the solutions are rational or not. By contrast, Vela, who is a true scholar, and I to a much lesser degree, would like the techniques we use to make sense and to be founded on fundamental principles. In this chapter I would like to explain the ideas of universal models for statistics. The basic idea is taken from recursive function theory, where a universal computer can imitate the actions of any special computer or program and execute them. Hence, given a class of models, each a parametric dis tribution, a universal model for the class is any parameter free distribution, which behaves like any model in the class on data generated by the model in the sense of assigning almost as large a probability or density to the data as the model. Unlike in recursive function theory the imitation is not exact except in the per symbol limit as the sample grows. We define this more precisely later. Universal models are new in statistics, although some of their uses are handled by the familiar concepts like consistency, estimator and criterion, all of which may in fact be replaced and generalized by universal models. In information theory the related constructs of universal data compression systems are ubiquitous and, one may say, most practical data compression systems are of that kind.
