Panacea or Poison: Can Propensity Score Modeling (PSM) Methods Replicate the Results from Randomized Control Trials (RCTs)?

Purpose: Criminal justice agencies rely on research to inform decisions related to policy and practice. Regardless of institutional focus (i.e., criminal victimization, law enforcement, corrections, and federal justice), identifying the practices that will best reduce crime and victimization requires ample research with strong methodology. Understood as the "gold standard" in research design, the randomized control trial (RCT) has been shown to provide reliable and valid findings. However, RCTs are often not feasible in many criminal justice settings because of ethical and practical considerations. As a substitute for RCTs, researchers have increasingly had to rely on other quasi-experimental designs and statistical techniques. One of the more popular techniques is propensity score modeling (PSM), which is designed to simulate the effects of an RCT. With the growing popularity and technological ease of using PSM, this research project seeks to address the critical question: Can PSM methods replicate the results from RCTs? Research Design and Methods: Using a selection of ten RCT databases available in the National Archive of Criminal Justice Data (NACJD), this study will employ five commonly used PSM techniques to these datasets including (1) one-to-one matching, (2) one-to-many matching, (3) optimal matching, (4) inverse probability of treatment weighting (IPTW), and (5) stratification on the propensity score. Analysis: For each of the datasets included in this study, a common metric effect size (ES) (i.e., Cohen's d) will be calculated along with a 95% confidence interval (Cl) for the five PSM techniques. The ES results will then be compared to those from the RCT. Following these individual comparisons, this project will also meta-analyze the compared differences in findings. The focus of our analyses will be on assessing the extent to which the Cls overlap with one another. This strategy will be instructive in determining whether there are significant differences found in the ES estimates of the various methodologies used. Products: The findings from this study will provide evidence to either support or revisit the reliance on PSM in criminal justice program and policy evaluations. This study will also help criminal justice agencies make more informed decisions about which types of research methodologies to invest their limited resources. As a result, multiple peer-reviewed publications are expected from each test of PSM and meta-analysis. Similarly, reports will be disseminated to NIJ and presented to state and federal agencies to determine how best to interpret and apply PSM related research findings. ca/ncf