Causal Inference and Comparative Analysis with Large-Scale Assessment Data

Causal Inference and Comparative Analysis with Large-Scale Assessment Data ................................................................................................. 522 Rubin Causal Model ........................................................................................... 522 Quasi-Experimental Designs ............................................................................. 524 Instrumental Variables ....................................................................................... 525

Introduction, Theory, and Assumptions ..................................................... 525 Estimation ........................................................................................................ 526 Other Considerations ..................................................................................... 528

Regression Discontinuity Designs .................................................................... 528 Introduction, Theory, and Assumptions ..................................................... 528 Estimation ........................................................................................................ 529 Other Considerations ..................................................................................... 531

Propensity Score Matching ................................................................................ 532 Introduction, Theory, and Assumptions ..................................................... 532 Estimation ........................................................................................................535 Other Considerations ..................................................................................... 537

Causal Mediation Analysis ................................................................................ 538 Scenarios of Causal and Comparative Analyses ............................................ 539

Scenario 1. An After-School Tutoring Program ..........................................540 Scenario 2. Comparing Achievement Scores for Students of Different Ancestries ...................................................................................540 Scenario 3. Cross-Ancestry Comparisons of After-School Program Effects ............................................................................................... 541

Conclusion ...........................................................................................................542 References .............................................................................................................542

Education researchers are increasingly interested in making causal inferences rather than simply describing correlations among variables. Drawing causal inferences from data hinges upon the design of the study, with experimental designs facilitating causal inferences most readily. However, oftentimes experimental designs are infeasible or impractical; moreover, researchers may want to make use of the vast amounts of publicly available large-scale secondary datasets. This chapter discusses the assumptions required for making causal inferences. Then, I provide an introduction to several techniques that may facilitate quasi-experimental designs when applied to secondary data, and in the process I note the specific sets of assumptions for inferring causality when using each technique. I conclude with a discussion of special considerations for researchers using international datasets, and I include examples where causal inferences may be possible. In cases where causal inferences are not possible, I discuss how researchers can use the techniques described in this chapter for careful comparative analyses.