Quality Assessment in Cochrane Reviews: do we practice what we preach?

Background: One of the most important steps in a systematic review (SR) is the critical appraisal of the quality of primary studies. Many studies have been published on the use of checklists and scales to assess methodological quality. The Cochrane Collaboration Handbook provides general principles that should be followed by the reviewers in an attempt to assess the quality of the primary studies. Nevertheless, there is still discussion about some issues such as: a) what is the most valid scale/checklist for the quality assessment ? b) what should be the use of quality assessment in the context of SRs? Objectives: The aim of this project is to review the approaches used in the quality assessment in Cochrane Reviews. Methods: A sample of SR's based on 50% of those published on the Cochrane Library Issue 1, 2002, stratified by Collaborative Review Groups (CRGs) and type of intervention (drugs, rehabilitation, prevention/screening, surgery/radiotherapy, intervention, communication/organisational/educational, other) were eligible for this study. Each CRG module and review has been reviewed by two people independently using two checklists specifically developed for this project. Results: Preliminary results based on a sample of 10% of all SRs show that ad hoc developed scales are the most frequently used tools for the assessment of the quality of studies, while 28% of the reviews use either Jadad or Schultz scales. The items that are most widely considered are: allocation concealment (78%) and completeness of follow-up (64%). Very often, an operational and reproducible definition of these items is not clearly reported in the methods nor in the methodological quality section. In 52% of the reviews items that are used do not correspond to items which were stated a priori in the methods section. 64 reviews (55%) do not report how they intend to use the results of the quality assessment and 13% of these will use it either as a criterion for study exclusion or for sensitivity analyses. In the majority of cases (69%) quality assessment is only reported in the "methodological quality section" and it is not linked to the results of the analyses. Conclusions: The results of this preliminary analysis show that despite the attention given to quality assessment of primary studies in the Cochrane Collaboration there is still substantial variation in the way quality of studies is assessed. Moreover, the way this assessment is eventually used in the analysis and interpretation of the results suffers from several inconsistencies. The poster will present full results and discuss the association between type of quality assessment and characteristics of the SR's. It will also address the opportunities for improving the consistency of Cochrane reviews in these important aspects.