Abstract
The mini-CEX is widely used to rate directly observed resident-patient encounters. Although several studies have explored the reliability of mini-CEX scores, the dimensionality of mini-CEX scores is incompletely understood. Objective: Explore the dimensionality of mini-CEX scores through factor analysis and generalizability analysis. Design: Factor analytic and generalizability study using retrospective data. Participants: Eighty five physician preceptors and 264 internal medicine residents (postgraduate years 1–3). Methods: Preceptors used the six-item mini-CEX to rate directly observed resident-patient encounters in internal medicine resident continuity clinics. We analyzed mini-CEX scores accrued over 4 years using repeated measures analysis of variance to generate a correlation matrix adjusted for multiple observations on individual residents, and then performed factor analysis on this adjusted correlation matrix. We also performed generalizability analyses. Results: Eighty-five preceptors rated 264 residents in 1,414 resident-patient encounters. Common factor analysis of these scores after adjustment for repeated measures revealed a single-factor solution. Cronbach’s alpha for this single factor (i.e. all six mini-CEX items) was ≥0.86. Sensitivity analyses using principal components and other method variations revealed a similar factor structure. Generalizability studies revealed a reproducibility coefficient of 0.23 (0.70 for 10 raters or encounters). Conclusions: The mini-CEX appears to measure a single global dimension of clinical competence. If educators desire to measure discrete clinical skills, alternative assessment methods may be required. Our approach to factor analysis overcomes the limitation of repeated observations on subjects without discarding data, and may be useful to other researchers attempting factor analysis of datasets in which individuals contribute multiple observations.
Similar content being viewed by others
References
Brennan, R. L. (2001). Generalizability theory. New York: Springer.
Cook, D. A., & Beckman, T. J. (2009). Does scale length matter? A comparison of nine- versus five-point rating scales for the mini-CEX. Advances in Health Sciences Education: Theory and Practice, 14, 655–664.
Cook, D. A., Dupras, D. M., Beckman, T. J., Thomas, K. G., & Pankratz, V. S. (2009). Effect of rater training on reliability and accuracy of mini-CEX scores: A randomized, controlled trial. Journal of General Internal Medicine, 24, 74–79.
Donato, A. A., Pangaro, L., Smith, C., Rencic, J., Diaz, Y., Mensinger, J., et al. (2008). Evaluation of a novel assessment form for observing medical residents: A randomised, controlled trial. Medical Education, 42, 1234–1242.
Durning, S. J., Cation, L. J., Markert, R. J., & Pangaro, L. N. (2002). Assessing the reliability and validity of the mini-clinical evaluation exercise for internal medicine residency training. Academic Medicine, 77, 900–904.
Fernando, N., Cleland, J., McKenzie, H., & Cassar, K. (2008). Identifying the factors that determine feedback given to undergraduate medical students following formative mini-CEX assessments. Medical Education, 42, 89–95.
Floyd, F. J., & Widaman, K. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7, 286–299.
Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
Greenburg, D. L., Durning, S. J., Cohen, D. L., Cruess, D., & Jackson, J. L. (2007). Identifying medical students likely to exhibit poor professionalism and knowledge during internship. Journal of General Internal Medicine, 22, 1711–1717.
Haber, R. J., & Avins, A. L. (1994). Do ratings on the American Board of internal medicine resident evaluation form detect differences in clinical competence? Journal of General Internal Medicine, 9, 140–145.
Harvill, L. M. (1991). NCME instructional module: Standard error of measurement. Educational Measurement: Issues and Practice, 10(2), 33–41.
Hatala, R., Ainslie, M., Kassen, B. O., Mackie, I., & Roberts, J. M. (2006). Assessing the mini-clinical evaluation exercise in comparison to a national specialty examination. Medical Education, 40, 950–956.
Herbers, J. E., Jr., Noel, G. L., Cooper, G. S., Harvey, J., Pangaro, L. N., & Weaver, M. J. (1989). How accurate are faculty evaluations of clinical competence? Journal of General Internal Medicine, 4, 202–208.
Hill, F., Kendall, K., Galbraith, K., & Crossley, J. (2009). Implementing the undergraduate mini-CEX: A tailored approach at Southampton University. Medical Education, 43, 326–334.
Hojat, M., Paskin, D. L., Callahan, C. A., Nasca, T. J., Louis, D. Z., Veloski, J., et al. (2007). Components of postgraduate competence: analyses of thirty years of longitudinal data. Medical Education, 41, 982–989.
Holmboe, E., Fiebach, N., Galaty, L., & Huot, S. (2001). Effectiveness of a focused educational intervention on resident evaluations from faculty. Journal of General Internal Medicine, 16, 427–434.
Holmboe, E. S., Huot, S., Chung, J., Norcini, J., & Hawkins, R. E. (2003). Construct validity of the miniclinical evaluation exercise (miniCEX). Academic Medicine, 78, 826–830.
Kogan, J. R., Bellini, L. M., & Shea, J. A. (2003). Feasibility, reliability, and validity of the mini-clinical evaluation exercise (mCEX) in a medicine core clerkship. Academic Medicine, 78(10 Suppl), S33–S35.
Kogan, J. R., Holmboe, E. S., & Hauer, K. E. (2009). Tools for direct observation and assessment of clinical skills of medical trainees: A systematic review. JAMA, 302, 1316–1326.
Kroboth, F. J., Hanusa, B. H., & Parker, S. C. (1996). Didactic value of the clinical evaluation exercise. Missed opportunities. Journal of General Internal Medicine, 11, 551–553.
Margolis, M. J., Clauser, B. E., Cuddy, M. M., Ciccone, A., Mee, J., Harik, P., et al. (2006). Use of the mini-clinical evaluation exercise to rate examinee performance on a multiple-station clinical skills examination: A validity study. Academic Medicine, 81(10 Suppl), S56–S60.
Nasca, T. J., Gonnella, J. S., Hojat, M., Veloski, J., Erdmann, J. B., Robeson, M., et al. (2002). Conceptualization and measurement of clinical competence of residents: A brief rating form and its psychometric properties. Medical Teacher, 24, 299–303.
Ney, E. M., Shea, J. A., & Kogan, J. R. (2009). Predictive validity of the mini-clinical evaluation exercise (mCEX): Do medical students’ mCEX ratings correlate with future clinical exam performance? Academic Medicine, 84(10 suppl), S21–S24.
Norcini, J. J., Blank, L. L., Arnold, G. K., & Kimball, H. R. (1995). The mini-CEX (clinical evaluation exercise): A preliminary investigation. Annals of Internal Medicine, 123, 795–799.
Norcini, J. J., Blank, L. L., Duffy, F. D., & Fortna, G. S. (2003). The mini-CEX: A method for assessing clinical skills. Annals of Internal Medicine, 138, 476–481.
O’Connor, B. P. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instrumentation, & Computers, 32, 396–402.
Shavelson, R. L., & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park: Sage Publications.
Sidhu, R. S., Hatala, R., Barron, S., Broudo, M., Pachev, G., & Page, G. (2009). Reliability and acceptance of the mini-clinical evaluation exercise as a performance assessment of practicing physicians. Academic Medicine, 84(10 suppl), S113–S115.
Silber, C. G., Nasca, T. J., Paskin, D. L., Eiger, G., Robeson, M., & Veloski, J. J. (2004). Do global rating forms enable program directors to assess the ACGME competencies? Academic Medicine, 79, 549–556.
Srinivasan, M., Hauer, K. E., Der-Martirosian, C., Wilkes, M., & Gesundheit, N. (2007). Does feedback matter? Practice-based learning for medical students after a multi-institutional clinical performance examination. Medical Education, 41, 857–865.
Thomas, P. A., Gebo, K. A., & Hellmann, D. B. (1999). A pilot study of peer review in residency training. Journal of General Internal Medicine, 14, 551–554.
Volkan, K., Simon, S. R., Baker, H., & Todres, I. D. (2004). Psychometric structure of a comprehensive objective structured clinical examination: A factor analytic approach. Advances in Health Sciences Education: Theory and Practice, 9, 83–92.
Weller, J. M., Jolly, B., Misur, M. P., Merry, A. F., Jones, A., Crossley, J. G. M., et al. (2009). Mini-clinical evaluation exercise in anaesthesia training. British Journal of Anaesthesia, 102, 633–641.
Wilkinson, T. J., & Frampton, C. M. (2003). Assessing performance in final year medical students. Can a postgraduate measure be used in an undergraduate setting? Medical Education, 37, 233–240.
Acknowledgments
Financial support
No external funding.
Authorship
All authors were involved in the planning and execution of this study and in the drafting and revising of this manuscript.
Ethical approval
Judged exempt by our Institutional Review Board.
Conflicts of interest statement
The authors have no affiliation with an organization with a financial interest in the subject matter, and are not aware of any conflicts of interest.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
Part 1. Factor analysis accounting for multiple observations on each resident
Code is for SAS 9.1. Original dataset for the post-workshop data is “cex_post” with data columns:
-
fac_id (unique preceptor identification code),
-
res_id (unique resident identification code),
-
rep_id (unique encounter identifier for preceptor-resident pairs with more than one observation; if preceptor A observed resident B three times, rep_id values would be 1 for the first encounter, 2 for the second, and 3 for the third)
-
ratings for each mini-CEX domain: couns, ex, human, hx, judg, org.
Step 1. Create an adjusted correlation matrix using mixed linear models with repeated measures on preceptors and residents.
-
(a)
First reformat the dataset for proc mixed:
-
(b)
Then create the adjusted correlation matrix. Note that you must be careful to determine the order of the variables in the resultant matrix (the item labels are not part of the matrix output).
-
(c)
The results of this analysis will appear as output on the screen. Again, it is essential to correctly identify which variable matches with each column in this matrix (column order is the same as the order of the variables in the parent dataset, in this case “cex_mixed”).
Step 2. Create a correlation data set for subsequent analysis
The values derived above can be manually used to create a data set of type “CORR” for subsequent analysis as shown below. Means and standard deviations can be determined using proc means. We estimated N (effective sample size) from the number of discrete preceptor-resident pairs (which can be found by counting the number of observations with rep_id = 1).
Step 3. Perform factor analysis on this adjusted correlation matrix:
Part 2. Additional tables: principal components factor analysis and adjusted/unadjusted correlation matrices
Rights and permissions
About this article
Cite this article
Cook, D.A., Beckman, T.J., Mandrekar, J.N. et al. Internal structure of mini-CEX scores for internal medicine residents: factor analysis and generalizability. Adv in Health Sci Educ 15, 633–645 (2010). https://doi.org/10.1007/s10459-010-9224-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-010-9224-9