ABSTRACT
We live in an era when politicians are anxious to claim that their policies are not based on ideologies, or simple opinions, but instead on evidence. Despite the claims of the more postmodern theorists that attempts to examine evidence in a rationalistic manner are futile, the claims of policymakers have never been more modernistic, or more concerned to show that they are implementing ‘what works’ (Davies, Nutley, and Smith, 1999). However, in practice, evaluating whether existing policies or programmes work, or present-
ing evidence that reforms are based on the best available research to achieve evidence-based policy, have proven difficult to achieve. In the UK, the claims by Secretary of State Andrew Lansley that the 2011 health reforms were evidence-based were met in some quarters of the informed media with derision (Goldacre, 2011). This suggests that policymakers’ understanding of what counts as evidence might be very different from those engaged in research, and points to important questions about both what should count as good evidence in policymaking, and how we should evaluate existing research to the end of producing better policy. When considering evidence, there is an obvious appeal in claiming to be taking a scientific
approach to the construction, assessment and evaluation of data. The problem is working out exactly what this means. Economists advocate cost-benefit approaches as a scientific approach, but such appraisal exercises have become parts of evaluation packages rather than being the entire approach. Instead, the foundation of evaluation studies has become based on experimental methods derived from natural science, and especially from the most applied of the scientific disciplines, medicine. This chapter considers the questions of what counts as evidence in evaluation, exploring three
approaches that attempt to provide a method for the synthesis of research, with each either attempting to provide an underpinning for the way that either interventions can be evaluated, or existing research synthesised for policy purposes (or both). It considers the strengths and weaknesses of each, as well as the assumptions they hold about what counts as good research. It concludes by suggesting that the realistic approach suggesting by Ray Pawson has the greatest potential for informing policy through research because of its more sophisticated consideration of the problems of applying social research to policy ends, but that its application and widespread usage will require a new mind-set from both policymakers and academics that recognises the limitations of what research can achieve, and which gives a greater role for creativity in social research. The chapter proceeds as follows: first it considers what many social researchers regard as the
gold standard in terms of evidence in evaluation, the randomised controlled trial (RCT), and
assesses the extent to which research outside of the hard-science paradigm meets the preconditions necessary for the application of RCT-based ideas. It then moves on to attempts by social researchers, especially those who are often engaged in asking questions with both clinical and qualitative elements, to make use of frameworks inspired by the RCT, adapting the criteria, often to try and include qualitative data, before finally moving onto a third approach presented by Ray Pawson and his collaborators which is based on very different assumptions to the RCT. Having considered the strengths and weaknesses of each evaluation approach, the chapter then moves on to its conclusion.
