Advances in Health Sciences Education

, Volume 18, Issue 2, pp 181–192

Understanding self-assessment as an informed process: residents’ use of external information for self-assessment of performance in simulated resuscitations

  • Jennifer L. Plant
  • Mark Corden
  • Michelle Mourad
  • Bridget C. O’Brien
  • Sandrijn M. van Schaik
Article

Abstract

Self-directed learning requires self-assessment of learning needs and performance, a complex process that requires collecting and interpreting data from various sources. Learners’ approaches to self-assessment likely vary depending on the learner and the context. The aim of this study was to gain insight into how learners process external information and apply their interpretation of this information to their self-assessment and learning during a structured educational activity. The study combined quantitative performance data with qualitative interview data. Pediatric residents led video-recorded simulated resuscitations and rated their crisis resource management skills on a validated 6-item instrument. Three independent observers rated the videos using the same instrument. During semi-structured interviews, each resident reviewed the video, rerated performance, discussed the self-assessment process, and interpreted feedback and observer scores. Transcripts were analyzed for themes. Sixteen residents participated. Residents’ self-assessed scores ranged widely but usually fell within two points of the observers. They almost universally lowered their scores when self-assessing after the video review. Five major themes emerged from qualitative analysis of their interviews: (1) residents found self-assessment important and useful in certain contexts and conditions; (2) residents varied in their self-directed learning behaviors after the simulated resuscitation; (3) quantitative observer assessment had limited usefulness; (4) video review was difficult but useful; and (5) residents focused on their weaknesses and felt a need for constructive feedback to enhance learning. The residents in our study almost uniformly embraced the importance of self-assessment for all medical professionals. Even though video review had a negative impact on their self-assessment scores and was perceived as painful, residents saw this as the most useful aspect of the study exercises residents. They were less accepting of the quantitative assessment by observers. Residents explained their tendency to focus on weaknesses as a way to create an incentive for learning, demonstrating that self-assessment is closely linked to self-directed learning. How learners can use video review and external assessment most effectively to guide their self-directed learning deserves further study.

Keywords

Crisis resource management skills Disconfirming feedback External assessment Informed self-assessment Self-directed learning Simulated resuscitation Video review 

References

  1. ABMS. (2006). Maintenance of certification. Retrieved April 19, 2009, from http://www.abms.org/Maintenance_of_Certification/ABMS_MOC.aspx.
  2. ACGME. (2007). Common program requirements: General competencies. Retrieved April 16, 2009, from http://www.acgme.org/outcome/comp/GeneralCompetenciesStandards21307.pdf.
  3. Barnsley, L., Lyon, P. M., Ralston, S. J., Hibbert, E. J., Cunningham, I., Gordon, F. C., et al. (2004). Clinical skills in junior medical officers: A comparison of self-reported confidence and observed competence. Medical Education, 38(4), 358–367.CrossRefGoogle Scholar
  4. Biernat, K., Simpson, D., Duthie, E. Jr, Bragg, D., & London, R. (2003). Primary care residents self assessment skills in dementia. Advances in Health Sciences Education : Theory and Practice, 8(2), 105–110.CrossRefGoogle Scholar
  5. Boud, D. (1999). Avoiding the traps: Seeking good practice in the use of self assessment and reflection in professional courses. Social Work Education, 18, 121–132.CrossRefGoogle Scholar
  6. Boyatzis, R. E. (1998). Transforming qualitative information: Thematic analysis and code development. Thousand Oaks: Sage Publications.Google Scholar
  7. Claridge, J. A., Calland, J. F., Chandrasekhara, V., Young, J. S., Sanfey, H., & Schirmer, B. D. (2003). Comparing resident measurements to attending surgeon self-perceptions of surgical educators. American Journal of Surgery, 185(4), 323–327.CrossRefGoogle Scholar
  8. Colthart, I., Bagnall, G., Evans, A., Allbutt, H., Haig, A., Illing, J., et al. (2008). The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10. Medical Teacher, 30(2), 124–145.CrossRefGoogle Scholar
  9. Creswell, J. W., & Clark, V. L. P. (2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks: Sage Publications.Google Scholar
  10. Davis, D. A., Mazmanian, P. E., Fordis, M., Van Harrison, R., Thorpe, K. E., & Perrier, L. (2006). Accuracy of physician self-assessment compared with observed measures of competence: A systematic review. JAMA, 296(9), 1094–1102.CrossRefGoogle Scholar
  11. Eppich, W. J., Adler, M. D., & McGaghie, W. C. (2006). Emergency and critical care pediatrics: Use of medical simulation for training in acute pediatric emergencies. Current Opinion in Pediatrics, 18(3), 266–271.CrossRefGoogle Scholar
  12. Epstein, R. M., Siegel, D. J., & Silberman, J. (2008). Self-monitoring in clinical practice: A challenge for medical educators. The Journal of Continuing Education in the Health Professions, 28(1), 5–13.CrossRefGoogle Scholar
  13. Eva, K. W., & Regehr, G. (2005). Self-assessment in the health professions: A reformulation and research agenda. Academic Medicine, 80(10 Suppl), S46–S54.CrossRefGoogle Scholar
  14. Eva, K. W., Armson, H., Holmboe, E., Lockyer, J., Loney, E., Mann, K., et al. (2012). Factors influencing responsiveness to feedback: On the interplay between fear, confidence, and reasoning processes. Advances in Health Sciences Education: Theory and Practice, 17(1), 15–26.CrossRefGoogle Scholar
  15. Fanning, R. M., & Gaba, D. M. (2007). The role of debriefing in simulation-based learning. Simulation in Healthcare, 2(2), 115–125.CrossRefGoogle Scholar
  16. Gaba, D. M., Fish, K. J., & Howard, S. K. (1994). Crisis management in anesthesiology. New York: Churchill-Livingstone.Google Scholar
  17. Jones, R., Panda, M., & Desbiens, N. (2008). Internal medicine residents do not accurately assess their medical knowledge. Adv Health Sci Educ Theory Pract, 13(4), 463–468.CrossRefGoogle Scholar
  18. Kim, J., Neilipovitz, D., Cardinal, P., Chiu, M., & Clinch, J. (2006). A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: The University of Ottawa Critical Care Medicine, High-Fidelity Simulation, and Crisis Resource Management I Study. Critical Care Medicine, 34(8), 2167–2174.CrossRefGoogle Scholar
  19. Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284.CrossRefGoogle Scholar
  20. Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121–1134.CrossRefGoogle Scholar
  21. Leopold, S. S., Morgan, H. D., Kadel, N. J., Gardner, G. C., Schaad, D. C., & Wolf, F. M. (2005). Impact of educational intervention on confidence and competence in the performance of a simple surgical task. The Journal of Bone and Joint Surgery American, 87(5), 1031–1037.CrossRefGoogle Scholar
  22. Mann, K., van der Vleuten, C., Eva, K., Armson, H., Chesluk, B., Dornan, T., et al. (2011). Tensions in informed self-assessment: How the desire for feedback and reticence to collect and use it can conflict. Academic Medicine, 86(9), 1120–1127.CrossRefGoogle Scholar
  23. Mort, J. R., & Hansen, D. J. (2010). First-year pharmacy students’ self-assessment of communication skills and the impact of video review. American Journal of Pharmaceutical Education, 74(5), 1–7, Article 78.CrossRefGoogle Scholar
  24. Parker, R. W., Alford, C., & Passmore, C. (2004). Can family medicine residents predict their performance on the in-training examination? Family Medicine, 36(10), 705–709.Google Scholar
  25. Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks: Sage Publishers.Google Scholar
  26. Sargeant, J. (2008). Toward a common understanding of self-assessment. The Journal of Continuing Education in the Health Professions, 28(1), 1–4.CrossRefGoogle Scholar
  27. Sargeant, J., Mann, K., van der Vleuten, C., & Metsemakers, J. (2008). “Directed” self-assessment: practice and feedback within a social context. The Journal of Continuing Education in the Health Professions, 28(1), 47–54.CrossRefGoogle Scholar
  28. Sargeant, J., Mann, K. V., van der Vleuten, C. P., & Metsemakers, J. F. (2009). Reflection: A link between receiving and using assessment feedback. Advances in Health Sciences Education: Theory and Practice, 14(3), 399–410.CrossRefGoogle Scholar
  29. Sargeant, J., Armson, H., Chesluk, B., Dornan, T., Eva, K., Holmboe, E., et al. (2010). The processes and dimensions of informed self-assessment: A conceptual model. Academic Medicine, 85(7), 1212–1220.CrossRefGoogle Scholar
  30. Sargeant, J., Eva, K. W., Armson, H., Chesluk, B., Dornan, T., Holmboe, E., et al. (2011). Features of assessment learners use to make informed self-assessments of clinical performance. Medical Education, 45(6), 636–647.CrossRefGoogle Scholar
  31. Stemler, S. E., & Tsai, J. (2008). Best practices in interrater reliability: three common approaches. In J. Osborne (Ed.), Best practices in quantitative methods. Thousand Oaks: Sage Publications.Google Scholar
  32. Tuckman, B. W. (1999). Conducting educational research (5th ed.). New York: Harcourt Brace College Publishers.Google Scholar
  33. Ward, M., MacRae, H., Schlachta, C., Mamazza, J., Poulin, E., Reznick, R., et al. (2003). Resident self-assessment of operative performance. American Journal of Surgery, 185(6), 521–524.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2012

Authors and Affiliations

  • Jennifer L. Plant
    • 1
  • Mark Corden
    • 2
  • Michelle Mourad
    • 2
  • Bridget C. O’Brien
    • 2
  • Sandrijn M. van Schaik
    • 2
  1. 1.Department of PediatricsUniversity of California DavisSacramentoUSA
  2. 2.University of California San FranciscoSan FranciscoUSA

Personalised recommendations