Nonrandomized quality improvement intervention trials might overstate the strength of causal inference of their findings

Objective: To assess the strength of causal inferences reported in randomized and nonrandomized evaluations of quality improvement (QI) interventions in relation to the study design and the direction of results for the primary outcomes. Study Design and Setting: We searched 11 journals for QI intervention studies that aimed to change clinician behavior. Statements that addressed the causal inference between intervention and outcomes were extracted and were rated by 34 researchers for the strength of causality. Results: We found 38 randomized controlled trials (RCTs) and 35 non-RCTs, and extracted 68 quotes from the abstracts and 139 from the main text. A significant interaction was found between study design and direction of results for the abstract quotes (P = 0.022). The ratings for non-RCTs were higher when the results were mixed, but for RCTs, they were higher if the results were positive or no effect, although none of the differences were statistically significant at alpha = 0.05 after adjusting for multiple comparisons. For the main text quotes, the causality rating was higher by 0.43 for RCTs than for non-RCTs after adjusting for the direction of results (P < 0.001). Conclusion: Authors might have overstated the strength of causal inference in the abstracts of non-RCTs, but appeared to report causality appropriately in the main text.