Meta-Analysis 3: Larger-Scale Analyses
In Chapter 7 I used forest plots to introduce meta-analysis and described how meta-analysis can summarize messy sets of results and find answers to important questions. In Chapter 8 I described two models and argued that we should almost always choose the random effects model. In this chapter I’ll discuss larger meta-analysis projects. The main messages are, first, that doing a large meta-analysis is a big job with many issues to consider and, second, that such an analysis can make an enormous contribution, and even answer research questions beyond the questions addressed by any of the individual studies. That’s more meta-analysis magic. Here’s the agenda:
• Seven steps to conducting a full meta-analysis • Finding moderators that account for ES variation • Analyzing more than one ES measure • Correcting for measurement error • Assessing publication bias • Meta-analysis becomes mainstream-the Cochrane and
Anyone who publishes a high-quality large-scale meta-analysis should, in my opinion, receive a gold medal, a large promotion, and a long, fully paid vacation. Such a research synthesis can be an immensely valuable scholarly contribution that brings order to confusion, helps set a future research agenda, and at the same time gives the best evidence-based practical advice. Such a project may take a year or more of work by a team, depending on the range of questions asked and the size and complexity of the relevant research literature. In this chapter I’ll outline the steps to undertaking a large-scale meta-analysis and issues to consider. It’s a daunting list, but I don’t want to be discouraging: Graduate students are increasingly carrying out a small to medium meta-analysis as the first
part of their research. Yes, a meta-analysis can take anything from five minutes to more than a year of effort, but somewhere in that vast range may be a project that’s right for you. Even if there’s already a good metaanalysis on your topic of interest, perhaps you can update and improve it. I outline below the seven main steps in conducting a large-scale meta-
analysis. I’ve chosen two meta-analytic review articles to illustrate the steps. The first investigates the effectiveness of acupuncture, and I’ll mention aspects of that study as I describe the steps. Then I’ll outline my second example review, about how we perceive the person we love,
also in terms of the seven steps. My steps largely follow a book by Harris Cooper (2010), which I highly recommend. Use it as the text for a reading seminar to
help you and some like-minded people develop your understanding of meta-analysis, or as a guide to carrying out your own meta-analysis. At every step there are choices to be made, based on your research goals, knowledge of the research area, and understanding of research methods and meta-analysis. Research synthesis demands creativity and scholarly wisdom; it’s no mere mechanical exercise. In the descriptions below of my two examples, note that I’m being very selective and including only small parts of what are two substantial reviews.