ABSTRACT
A time-honoured approach to informing decisions is to seek out an expert opinion. Appealing to expertise has intuitive appeal; most of us would prefer to act on the medical advice of a qualified doctor than rely on the judgement of our friends or neighbours. There are difficulties with the expert model, however. Whereas in some cases it may be trivial to identify the right person – their background or specialist training might mark them out for the role – in many other cases identifying a true expert is challenging, and those who appear to be expert in a matter may possess little actual expertise. Tetlock (2005) describes a 20-year study in which several hundred experts drawn from many fields, from professors to journalists, were asked to make thousands of predictions about the future. The predictions turned out to be only marginally more accurate than chance, and the most recognized experts – those who advise governments and business leaders, appear on television and are regularly quoted in newspapers – performed particularly poorly. Even when a real expert has been located, simply asking for an
opinion may not be the best way to ensure an honest response. Experts can suffer conflicts of interest: they may desire to tell us what they think we’d like to hear (Prendergast, 1993), or feel it is safer to herd with other experts (to avoid being wrong when everyone else is right).4 A narrow focus on conventional circles of expertise might cause valuable knowledge to be missed; recent work has shown that official forecasts of unemployment can be improved using insights from internet search data (D’Amuri and Marcucci, 2009). Similar results have emerged for the detection of influenza (Ginsberg et al., 2009; Polgreen et al., 2007). Instead of relying on a single, potentially biased, opinion we might think about consulting several experts and somehow pooling their opinions. This may go some way towards mitigating biases, although it still leaves open the problem of distinguishing true experts. It also raises the non-trivial question of which pooling mechanism should be used.5
