ABSTRACT
The aim of this study was to establish, using relevant randomized control trial (RCT) simulation, the accuracy of reverse propensity score (RPS)–based testing for second-order selection bias effect. RCTs with dichotomous outcomes, varying parameters and without effect size differences between intervention groups were simulated for one biased and one non-biased scenario. In each, RPS-based testing, using logistic regression, was conducted at alpha levels 1%, 5% and 20%. For each alpha level, the pooled test specificity and sensitivity (95% confidence intervals [CIs]) were established and summary receiver operating characteristic (SROC) curves were generated. Diagnostic odds ratios (DOR, 95% CI) were computed for different RCT parameter settings. The simulated bias effected an overestimation of the true odds ratio (OR = 1.00) to a range between OR 0.00 and 0.04. For each scenario, 1575 test runs were conducted. The pooled test sensitivity was 0.85 (95% CI: 0.83–0.86), 0.92 (95% CI: 0.90–0.93) and 0.98 (95% CI: 0.97–0.98) for alpha 1%, 5% and 20%; and the pooled test specificity was 0.91 (95% CI: 0.90–0.93), 0.80 (95% CI: 0.78–0.82) and 0.60 (95% CI: 0.57–0.62) for alpha 1%, 5% and 20%, respectively. SROC curves indicated highest overall test accuracy at alpha 1%. DOR values were highest with the maximal procedure as randomisation method; lower block size; higher sample size; higher bias. RPS-based testing may detect second-order selection bias effect in RCTs with high accuracy.
