Abstract
Residency educators are identifying approaches to measure resident competence. Portfolios are well suited since they require work already completed as part of patient care where competency must be demonstrated. This paper describes assessment of the reliability and validity of portfolios in a psychiatry residency program. This was a cross-sectional study across 4years of residency education. Using guidelines, 18 residents assembled portfolios containing five entries chosen from 13 skills. Trained raters scored the portfolios. Residents and faculty were interviewed about their perceptions. Generalizability results indicated five entries and two raters were sufficient for relative decisions. Six entries or a third rater would be sufficient for absolute decisions. Portfolio scores tended to improve with years of training and correlated with psychiatric knowledge but not clinical performance. Residents and faculty identified benefits to assembling a portfolio. Portfolios incorporate tasks embedded in the residency to provide evidence of resident competency. The results support that the score is reliable and valid.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Cogbill, K.K., O’Sullivan, P.S. & Clardy, J. (in press). Residents’ perceptions of effectiveness of twelve evaluation methods for measuring competency. Academic Psychiatry.
M.H. Davis M. Friedman Ben-David R.M. Harden P. Howie J. Ker C. McGhee M.J. Pippard D. Snadden (2001) ArticleTitlePortfolio assessment in medical students’ final examinations Medical Teacher 23 357–366
L.S. Feldt R.L. Brennan (1989) Reliability R L. Linn (Eds) Educational Measurement EditionNumber3 American Council on Education/Macmillan New York 105–146
M. Friedman Ben David M.H. Davis R.M. Harden P.W. Howie J. Ker M.J. Pippard (2001) ArticleTitleAMEE Medical Education Guide No. 24. Portfolios as a method student assessment Medical Teacher 23 535–551
P. Gallagher (2001) ArticleTitleAn evaluation of a standards based portfolio Nurse Education Today 21 409–416
F.T. Gannon P.R. Draper R. Watson S. Proctor I.J. Norman (2001) ArticleTitlePutting portfolios in their place Nurse Education Today 21 534–540
E.S. Holmboe R.E. Hawkins (1998) ArticleTitleMethods for evaluating the clinical competence of residents in internal medicine Annals of Internal Medicine 129 42–48
M.K Lettus P.H. Moessner L. Dooley (2001) ArticleTitleThe clinical portfolio as an assessment tool Nursing Administration Quarterly 25 74–79
J.J. Norcini L.L. Blank G.K. Arnold H.R. Kimball (1995) ArticleTitleThe mini-CEX (clinical evaluation exercise): A preliminary investigation Annals of Internal Medicine 123 795–799
P.S. O ‘Sullivan K. Cogbill C. McClain J. Clardy M. Reckase (2002) ArticleTitlePortfolios as a novel approach for residency evaluations Academic Psychiatry 26 173–179
J. Pitts C. Coles P. Thomas F. Smith (2002) ArticleTitleEnhancing reliability in portfolio assessment: Discussions between assessors Medical Teacher 24 197–201
M.D. Reckase (1995) ArticleTitlePortfolio assessment: A theoretical estimate of score reliability Educational Measurement: Issues and Practice 14 12–31
D. Snadden M. Thomas (1998) ArticleTitleThe use of portfolio learning in medical education Medical Teacher 20 192–208
S.R. Swing (2002) ArticleTitleAssessing the ACGME general competencies: General considerations and assessment methods Academic Emergency Medicine 9 1278–1288
L.S. Wenzel K.L. Briggs B.L. Puryear (1998) ArticleTitlePortfolio: Authentic assessment in the age of the curriculum revolution Journal of Nursing Education 37 208–212
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
O’sullivan, P.S., Reckase, M.D., McClain, T. et al. Demonstration of Portfolios to Assess Competency of Residents. Adv Health Sci Educ Theory Pract 9, 309–323 (2004). https://doi.org/10.1007/s10459-004-0885-0
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/s10459-004-0885-0