A large-scale study of the reliabilities of checklist scores and ratings of interpersonal and communication skills evaluated on a standardized-patient examination

Abstract

Checklists and rating scales are both used for assessing examinees in standardized-patient (SP) examinations. A common presumption, is that checklists are more objective than and hence superior to rating scales with respect to psychometric properties such as reliability. Recently, this presumption has been questioned and studies have been cited which support the clain that “objectified methods do not inherently provide more reliable scores.” The purpose of this study was to further explore this issue, using a specially designed checklist/rating scale form for assessing interpersonal and communication skills.

Methods. A 26-item checklist/rating scale form was developed, which consisted of five sections of 3 to 7 checklist items each, with a space provided at the end of each section for the SP to make a single global rating of the items in that section. Analyses were performed on data for 1,048 fourth-year medical students in the eight schools in the New York City Consortium, tested on the 7-case SP examination administered at The Morchand Center of Mount Sinai School of Medicine.

Results/Conclusion. The intercase reliabilities for the checklist scores and ratings, respectively, were: 0.33 and 0.39 for eliciting information, 0.33 and 0.39 for non-verbal behavior, 0.55 and 0.52 for patient education, 0.48 and 0.45 for professional manner, and 0.49 and 0.52 for patient satisfaction. The correlations between the checklist scores and ratings for the five sections were 0.66, 0.60, 0.80, 0.69, and 0.75, respectively. The pattern of these results provides further support for the claim that “objectified” measures, like checklists, “do not inherently provide more reliable scores.”