Competency Assessment

  • Anjan Shah
  • Samuel DeMariaEmail author
  • Andrew Goldberg
Part of the Comprehensive Healthcare Simulation book series (CHS)


The paradigm shift within medical education to one that is outcome based has made competency assessment critically important. Currently, residents are expected to meet and exceed specific core competencies prior to completion of training. Furthermore, attending physicians have a litany of challenges in maintaining their training after graduation. Given the multitude of practitioners requiring assessment in some phase of training, it is apparent the number of challenges that exist in creating a framework for competency assessment that will work for everyone. One such tool that can be utilized to create a fully manipulatable assessment that will work for varying levels of practitioners is human simulation, coupled with a clear guide or protocol detailing what observable behaviors are expected. A low- or high-fidelity simulated environment, using a myriad of personnel and devices, can ensure patient safety and at the same time allows for manipulation of variables to guarantee consistent delivery of scenarios to those being assessed. Competency assessment in anesthesiology can and should be used to ensure adequate introduction and preparation to monitor the development and progression of both technical and nontechnical skills, as well as for attending physicians to ensure proper maintenance of licensure to retraining of physicians to reenter practice.


Healthcare simulation Competency assessment Competency assessment in anesthesiology Simulation-based assessment Anesthesiology Patient safety Retraining of physicians Anesthesiology physician certification Maintenance of certification in anesthesiology 


  1. 1.
    Goldberg A, et al. High-stakes simulation-based assessment for retraining and returning physicians to practice. Int Anesthesiol Clin. 2015;53(4):70–80.PubMedCrossRefGoogle Scholar
  2. 2.
    McGaghie WC, et al. A critical review of simulation-based medical education research: 2003–2009. Med Educ. 2010;44(1):50–63.CrossRefGoogle Scholar
  3. 3.
    Issenberg SB, et al. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach. 2005;27(1):10–28.PubMedPubMedCentralCrossRefGoogle Scholar
  4. 4.
    Weller JM, et al. Psychometric characteristics of simulation-based assessment in anaesthesia and accuracy of self-assessed scores. Anaesthesia. 2005;60(3):245–50.PubMedCrossRefGoogle Scholar
  5. 5.
    Cook DA. When I say... validity. Med Educ. 2014;48(10):948–9.PubMedCrossRefGoogle Scholar
  6. 6.
    Cook DA, et al. A contemporary approach to validity arguments: a practical guide to Kane’s framework. Med Educ. 2015;49(6):560–75.PubMedCrossRefGoogle Scholar
  7. 7.
    Kane MT. Validating the interpretations and uses of test scores. J Educ Meas. 2013;50(1):1–73.CrossRefGoogle Scholar
  8. 8.
    Gingerich A, Regehr G, Eva KW. Rater-based assessments as social judgments: rethinking the etiology of rater errors. Acad Med. 2011;86(10 Suppl):S1–7.PubMedCrossRefGoogle Scholar
  9. 9.
    Yeates P, et al. Seeing the same thing differently: mechanisms that contribute to assessor differences in directly-observed performance assessments. Adv Health Sci Educ Theory Pract. 2013;18(3):325–41.PubMedCrossRefGoogle Scholar
  10. 10.
    Accreditation Council for Graduate Medical Education. Next Accreditation System (NAS) and Milestones – a resident perspective. 2014 [cited 2016 09/25]. Available from:
  11. 11.
    The Accreditation Council for Graduate Medical Education and The American Board of Anesthesiology. The Anesthesiology Milestone Project. 2015 [cited 2016 09/20]. Available from:
  12. 12.
    Accreditation Council for Graduate Medical Education and The American Board of Medical Specialties. ACGME competencies: suggested best methods for evaluation (Toolbox of assessment methods). 2000 [cited 2016 09/26]. Available from:
  13. 13.
    Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 Suppl):S63–7.PubMedCrossRefGoogle Scholar
  14. 14.
    Pangaro L. A new vocabulary and other innovations for improving descriptive in-training evaluations. Acad Med. 1999;74(11):1203–7.PubMedCrossRefGoogle Scholar
  15. 15.
    Dreyfus, S.E. A five-stage model of the mental activities involved in directed skill acquisition. 1980 [cited 2016 09/28]. Monograph available online. Available from:
  16. 16.
    Eraut M. Developing professional knowledge and competence. London: RoutledgeFalmer; 1994. p. 124.Google Scholar
  17. 17.
    Barrows HS, Abrahamson S. The programmed patient: a technique for appraising student performance in clinical neurology. J Med Educ. 1964;39:802–5.PubMedGoogle Scholar
  18. 18.
    Abrahamson S, Denson JS, Wolf RM. Effectiveness of a simulator in training anesthesiology residents. J Med Educ. 1969;44(6):515–9.PubMedGoogle Scholar
  19. 19.
    Kane MT. An argument-based approach to validity. Psychol Bull. 1992;112(3):527–35.CrossRefGoogle Scholar
  20. 20.
    Downing SM, Haladyna TM. Validity and its threats. In: Downing SM, Yudkowsky R, editors. Assessment in health professions education. New York: Routledge; 2009. p. 21–56.CrossRefGoogle Scholar
  21. 21.
    Goldmann K, Steinfeldt T. Acquisition of basic fiberoptic intubation skills with a virtual reality airway simulator. J Clin Anesth. 2006;18(3):173–8.PubMedCrossRefGoogle Scholar
  22. 22.
    Maran NJ, Glavin RJ. Low- to high-fidelity simulation – a continuum of medical education? Med Educ. 2003;37(Suppl 1):22–8.PubMedCrossRefGoogle Scholar
  23. 23.
    Demaria S Jr, et al. Adding emotional stressors to training in simulated cardiopulmonary arrest enhances participant performance. Med Educ. 2010;44(10):1006–15.PubMedCrossRefGoogle Scholar
  24. 24.
    USMLE. Bulletin of Information. 2016 [cited 2016 09/30]. Available from:
  25. 25.
    Baker KS, Cormican D, Seidman PA. Summer anesthesiology externship: demonstrating the ability of early clinical involvement to educate and increase specialty interest. J Educ Perioper Med. 2012;14(4):E063.PubMedPubMedCentralGoogle Scholar
  26. 26.
    Fletcher GC, et al. The role of non-technical skills in anaesthesia: a review of current literature. Br J Anaesth. 2002;88(3):418–29.PubMedCrossRefGoogle Scholar
  27. 27.
    Glavin RJ. Excellence in anesthesiology: the role of nontechnical skills. Anesthesiology. 2009;110(2):201–3.PubMedGoogle Scholar
  28. 28.
    Flynn FM, Sandaker K, Ballangrud R. Aiming for excellence – a simulation-based study on adapting and testing an instrument for developing non-technical skills in Norwegian student nurse anaesthetists. Nurse Educ Pract. 2016;22:37–46.PubMedCrossRefGoogle Scholar
  29. 29.
    Lyk-Jensen HT, et al. Assessing nurse anaesthetists’ non-technical skills in the operating room. Acta Anaesthesiol Scand. 2014;58(7):794–801.PubMedCrossRefGoogle Scholar
  30. 30.
    Wunder LL, et al. Objective structured clinical examination as an educational initiative for summative simulation competency evaluation of first-year student registered nurse anesthetists’ clinical skills. AANA J. 2014;82(6):419–25.PubMedGoogle Scholar
  31. 31.
    Wunder LL. Effect of a nontechnical skills intervention on first-year student registered nurse anesthetists’ skills during crisis simulation. AANA J. 2016;84(1):46–51.PubMedGoogle Scholar
  32. 32.
    Accreditation Council for Graduate Medical Education. Specialty-specific Duty Hour Definitions. 2012 [cited 2016 09/29]. Available from:
  33. 33.
    Mudumbai SC, et al. External validation of simulation-based assessments with other performance measures of third-year anesthesiology residents. Simul Healthc. 2012;7(2):73–80.CrossRefGoogle Scholar
  34. 34.
    Berkenstadt H, et al. Deficits in the provision of cardiopulmonary resuscitation during simulated obstetric crises: results from the Israeli Board of Anesthesiologists. Anesth Analg. 2012;115(5):1122–6.PubMedCrossRefGoogle Scholar
  35. 35.
    Ben-Menachem E, et al. Objective structured clinical examination-based assessment of regional anesthesia skills: the Israeli National Board Examination in anesthesiology experience. Anesth Analg. 2011;112(1):242–5.PubMedCrossRefGoogle Scholar
  36. 36.
    Flin R, et al. Anaesthetists’ non-technical skills. Br J Anaesth. 2010;105(1):38–44.PubMedCrossRefGoogle Scholar
  37. 37.
    Fehr JJ, et al. Simulation-based assessment of pediatric anesthesia skills. Anesthesiology. 2011;115(6):1308–15.PubMedGoogle Scholar
  38. 38.
    Blum RH, et al. A method for measuring the effectiveness of simulation-based team training for improving communication skills. Anesth Analg. 2005;100(5):1375–80, table of contentsPubMedCrossRefGoogle Scholar
  39. 39.
    Malec JF, et al. The mayo high performance teamwork scale: reliability and validity for evaluating key crew resource management skills. Simul Healthc. 2007;2(1):4–10.PubMedCrossRefGoogle Scholar
  40. 40.
    Blum RH, et al. Simulation-based assessment to identify critical gaps in safe anesthesia resident performance. Anesthesiology. 2014;120(1):129–41.PubMedCrossRefGoogle Scholar
  41. 41.
    Goldberg A, et al. Learning through simulated independent practice leads to better future performance in a simulated crisis than learning through simulated supervised practice. Br J Anaesth. 2015;114(5):794–800.PubMedCrossRefGoogle Scholar
  42. 42.
    Wayne DB, et al. A longitudinal study of internal medicine residents’ retention of advanced cardiac life support skills. Acad Med. 2006;81(10 Suppl):S9–S12.PubMedCrossRefGoogle Scholar
  43. 43.
    Wayne DB, et al. Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice. J Gen Intern Med. 2006;21(3):251–6.PubMedPubMedCentralCrossRefGoogle Scholar
  44. 44.
    Wayne DB, et al. Simulation-based education improves quality of care during cardiac arrest team responses at an academic teaching hospital: a case-control study. Chest. 2008;133(1):56–61.PubMedCrossRefGoogle Scholar
  45. 45.
    Rodgers DL, Securro S Jr, Pauley RD. The effect of high-fidelity simulation on educational outcomes in an advanced cardiovascular life support course. Simul Healthc. 2009;4(4):200–6.PubMedPubMedCentralCrossRefGoogle Scholar
  46. 46.
    The American Board of Anesthesiology. The American Board of Anesthesiology: APPLIED (Staged Exams). 2016 [cited 2016 09/30]. Available from:
  47. 47.
    The American Board of Anesthesiology. Applied Examination. [cited 2016 12/13/2016]. Available from:
  48. 48.
    The American Board of Anesthesiology. MOCA Part 4 Requirements. [cited 2016]. Available from:
  49. 49.
    Steadman RH, et al. Practice improvements based on participation in simulation for the maintenance of certification in anesthesiology program. Anesthesiology. 2015;122(5):1154–69.PubMedCrossRefGoogle Scholar
  50. 50.
    Levine AI, et al. Simulation-based Maintenance of Certification in Anesthesiology (MOCA) course optimization: use of multi-modality educational activities. J Clin Anesth. 2012;24(1):68–74.PubMedCrossRefGoogle Scholar
  51. 51.
    Grace ES, et al. Physicians reentering clinical practice: characteristics and clinical abilities. J Contin Educ Heal Prof. 2011;31(1):49–55.CrossRefGoogle Scholar
  52. 52.
    Jewett EA, Brotherton SE, Ruch-Ross H. A national survey of ‘inactive’ physicians in the United States of America: enticements to reentry. Hum Resour Health. 2011;9:7.PubMedPubMedCentralCrossRefGoogle Scholar
  53. 53.
    Gallagher CJ, Tan JM. The current status of simulation in the maintenance of certification in anesthesia. Int Anesthesiol Clin. 2010;48(3):83–99.PubMedCrossRefGoogle Scholar
  54. 54.
    DeMaria S Jr, Levine AI, Bryson EO. The use of multi-modality simulation in the retraining of the physician for medical licensure. J Clin Anesth. 2010;22(4):294–9.PubMedCrossRefGoogle Scholar
  55. 55.
    Levine AI, et al. Role of simulation in U.S. physician licensure and certification. Mt Sinai J Med. 2012;79(1):140–53.PubMedCrossRefGoogle Scholar
  56. 56.
    Steadman RH, Huang YM. Simulation for quality assurance in training, credentialing and maintenance of certification. Best Pract Res Clin Anaesthesiol. 2012;26(1):3–15.PubMedCrossRefGoogle Scholar
  57. 57.
    DeMaria S Jr, et al. Simulation-based assessment and retraining for the anesthesiologist seeking reentry to clinical practice: a case series. Anesthesiology. 2013;119(1):206–17.PubMedCrossRefGoogle Scholar
  58. 58.
    Ilgen JS, et al. A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Med Educ. 2015;49(2):161–73.PubMedCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of AnesthesiologyPerioperative and Pain Medicine, Icahn School of Medicine at Mount SinaiNew YorkUSA
  2. 2.Department of AnesthesiologyJames J Peters VA Medical CenterNew YorkUSA

Personalised recommendations