Assessment of communication skills and self-appraisal in the simulated environment : feasibility of multirater feedback with gap analysis

INTRODUCTION: Multirater assessment is a powerful means of measuring communication skills. The use of gap analysis to assess self-appraisal is a strength of this technique. On the basis of Kalamazoo Consensus Statement framework and 360-degree assessment models, we developed a multirater instrument with gap analysis, with the goals of examining both communication skills and situational self-appraisal, and assessing the feasibility of the combined approach. METHODS: The multirater communication skills instrument was used to assess Pediatric and Neonatal Intensive Care fellows after participation in seven simulated family meetings. Instrument reliability was determined using Cronbach's Alpha and Factorial Analysis. Correlations between rater groups were examined with Spearman's Rank Coefficient. Gap analyses and rater perceptions of the instruments were analyzed using descriptive statistics. RESULTS: Seven pediatric intensive care unit and neonatal intensive care fellows were each assessed by 11 to 18 raters (108 total assessments). Correlations were identified between disciplinary groups within each encounter. Among the 7 fellows, 30 communication strengths or areas needing improvement and 24 significant gaps were identified, indicating self under-appraisals, 9 (38%) of which overlapped. The instrument was logistically feasible and well received. CONCLUSIONS: Our multirater communication skills instrument with gap analysis proved useful in identifying areas of strength and areas needing improvement, and in highlighting areas of self over- and under-appraisal that require focused feedback. The use of multirater assessment with gap analysis, in a simulated and "safe" environment, may assist in the delivery of feedback to trainees.