"Do the conclusions look as good as they seem?" A review of quality improvement intervention studies

PURPOSE: To assess the appropriateness of conclusions reported in quality improvement (QI) intervention studies. RELEVANCE: This study assessed the quality of reporting of research that aimed to improve health professional practice behaviours, an area that is highly relevant to physical therapists and health care administrators. PARTICIPANTS: We assembled a 38-member expert panel to assess the strength of causal inference suggested by the causal statements. This panel consisted of clinical epidemiologists, clinical trial methodologists, health services researchers, and clinical researchers. Half of the panelists were English first-language speaking. METHODS: We hand-searched 11 major medical journals or health services research journals for randomized and non-randomized evaluations of QI interventions (RCTs, non-RCTs) published between January 2002 and December 2003. Eligible studies were those evaluating interventions that aimed to change health professional behaviours based on research evidence. Two independent reviewers extracted data for each trial, including study characteristics, methodology, and all statements addressing the causal effect between the intervention and outcomes in the abstract and the main text. The expert panel rated the strength of causality of each quote on a Likert scale (range 1-7, higher = stronger causal relationship), assuming that all quotes were from well designed RCTs. Each panelist rated 60 to 70 randomly assigned quotes and each quote was rated by 10 to 12 panelists. ANALYSIS: Two-way ANOVA was used to assess main effects and the interaction between study designs (randomized controlled trials (RCTs) versus non-RCTs) and results of primary outcomes (statistical significant and mixed results versus no effect) on causality ratings. RESULTS: 73/4543 studies met eligibility criteria (38 RCTs; 35 non-RCTs) and 207 causality quotes were extracted. Ratings were received from 34 panelists (response rate=89.5%). In studies where more than one quote was extracted, only the maximum score was used for the primary analysis (66 abstract quotes: RCTs = 34, non-RCTs = 32; 68 main text quotes: RCTs = 34, non-RCTs = 34). Among the abstract quotes, the mean causality rating was 4.16 (95% confidence interval [CI]: 3.90, 4.42) in RCTs and 5.10 (95% CI: 4.91, 5.30) in non-RCTs. The difference in adjusted mean causality rating (RCT – Non-RCT) was −0.114 (favoring non-RCTs; 95% CI: −0.354, 0.125). Among the main text quotes (RCTs rating = 4.83; 95% CI: 4.58, 5.09; non-RCTs = 5.27; 95% CI: 5.05, 5.48), the difference in the adjusted mean causality rating (RCT – non-RCT) was 0.370 (favoring RCTs; 95% CI: 0.153, 0.587). Both rater and study result were highly significant factors. We checked for interactions between study result and design and found only a borderline significant interaction with the abstracts (p = 0.028). CONCLUSIONS: We failed to find statistically significant difference in the reporting of causal relationships between non-RCTs and RCTs in abstract quotes; however, non-RCTs consistently scored higher than RCTs in the causality rating. The results suggest that quality improvement researchers may over emphasise causal inference in non-RCTs in the abstracts. IMPLICATIONS: Conclusions of studies that employed non-RCT designs can be misleading if authors are overzealous in stating the causal relationship. Our review is the first step in improving the appropriateness of conclusions stated in QI intervention studies.