Using AMSTAR to assess the methodological quality of systematic reviews: an external validation study

Background: AMSTAR is an instrument for the evaluation of systematic reviews’ (SRs) methodological quality of conduct, a pre-requisite for the interpretation and application of SRs findings. Previous studies suggested that AMSTAR is reliable, valid and easy to use. A broader range of assessors on different types of SRs is needed. Objectives: To measure the reliability, construct validity and feasibility of AMSTAR on a sample of SRs in different medical field. Methods: A convenient sample of SRs in different medical fields was independently assessed by two reviewers using: AMSTAR and the Overview of Quality Assessment Questionnaire (OQAQ). Reliability was assessed using the weighted Cohen’s kappa. Construct validity was investigated comparing the two instruments using Pearson’s Rank correlation coefficient. Time to complete scoring was also recorded. Results: Fifty-four SRs focusing on primary prevention of hypertension and colorectal cancer screening and treatment were assessed using both AMSTAR and OQAQ. Eleven out of 54 were Cochrane SRs. 78% of the SRs analysed included a meta-analysis. Preliminary data on half sample showed an overall score ranging from 0 to 11 (median: 6) for AMSTAR and from 0 to 9 (median: 6) for the OQAQ. The inter-observer agreement of the individual items on AMSTAR ranged from fair to very good with a kappa going from 0.20 to 0.92. Four items scored a kappa >0.75. Fair agreements related to items about methods to combine results and conflict of interest. The kappa for each individual item for OQAQ ranged from 0.25 to 0.81. The reliability of the total score was fair for both checklists: kappa scored 0.35 for AMSTAR and 0.32 for OQAQ. Final results will be presented. Conclusions: This study serves as further validation of the AMSTAR involving a broader range of assessors and SRs and could help to better refine the instruments and disseminate its use.