Reliability and accuracy in reporting CIN in 14 laboratories : developing new indices of diagnostic variability in an interlaboratory study

OBJECTIVE: To report the results of the second phase of a pilot study of an Italian national external quality evaluation program (EQA) aimed at assessing whether participation in the first phase had increased the reliability and accuracy of diagnoses and to what extent. STUDY DESIGN: In the second phase, two sets of cervical smears (similar to the ones examined in the first phase) were circulated among the 14 participating cytologic laboratories throughout Italy. Responses were recorded on a standardized form. Participants were asked to judge the adequacy of each smear and to formulate a diagnosis. They weve also asked to recommend management of the patient on the basis of the smear report and to evaluate the degree of diagnostic difficulty of each slide. The results were discussed in workshops, and it was possible to reach a consensus diagnosis on 37 of 40 smears. In the statistical analysis, new indices of diagnostic variability were developed and calculated; a gross index of agreement, unweighted and weighted kappas, analysis of exchangeability, sensitivity and specificity were also estimated. RESULTS: The results of the second phase are similar to those of the first phase and no substantial improvement in accuracy and little reduction in variability were observed. The interventions carried out in this study (discussion between representatives gf laboratories of diagnostic differences and reassessment of the most controversial slides) were aimed at increasing consensus among the participating pathologists but were insufficient to change the diagnostic routines in their laboratories. CONCLUSION: It may be advisable to promote two kinds of interlaboratory quality programs with two separate but integrated components: (1) a cove component with slides having clear-cut diagnoses, and (2) a continuing education component.