Skip to main content
SpringerLink
Log in

Assessment of Clinical Skills Using Simulator Technologies

  • Educational Assessment, Interventions, and Outcomes
  • Published:
Academic Psychiatry Aims and scope Submit manuscript

Abstract

Objective

Simulation technologies are used to assess and teach competencies through the provision of reproducible stimuli. They have exceptional utility in assessing responses to clinical stimuli that occur sporadically or infrequently. In this article, the authors describe the utility of emerging simulation technologies, and discuss critical issues in simulator-based skills assessment and appropriate results analysis.

Method

Based on literature search and expert consensus, the authors discuss three simulation technologies: standardized patients and the objective structured clinical examination; the integrated high fidelity mannequin; virtual clinical stations and the objective structured virtual examination.

Results

The authors explore the current state of these technologies: uses, cost, limitations, and likely future applications. For instance, tele-standardized patients may test learners’ communication/management approach to challenges during tele-consultation, such as a suicidal patient several hundred miles away. Integrated mannequins may test leadership skills during psychiatric emergencies. Case-based interactive virtual clinical assessment tools may test learners’ decision-making skills or self-reflection. However, these exciting tools must be implemented systematically. Specifically, educators must define the competencies of interest precisely. Appropriate data analysis will generate dependable results, ascribing the correct proportion of outcome variability to individual learner behavior. Careful analysis and utilization of results will allow justification of the costs to major stakeholders.

Conclusions

Simulation technologies offer exciting possibilities for skills evaluation and clinical practice improvement. When used creatively and appropriately, they form a useful adjunct in the armament of educators addressing the question, “Is this physician competent?”

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Srinivasan M, Hilty D: Teaching and helping others to learn, in Teaching and Learning in Academic Medicine: A Concise Guide for Academic Physicians. Arlington, Va, Academic Psychiatric Publishing, 2005

    Google Scholar 

  2. Keenan C, Srinivasan M: Teaching in ambulatory care, in Association of Program Directors in Internal Medicine Program Manual, 8th ed. Edited by Henderson MH. 2005

  3. Davis DA, Thomson MA, Oxman AD, et al: Changing physician performance: a systematic review of the effect of continuing medical education strategies. JAMA 1995; 274: 700–705

    Article  CAS  PubMed  Google Scholar 

  4. Rossi PH, Freeman HE, Lipsey MW: Evaluation: a systematic approach. Thousand Oaks, Calif, Sage Publications, 1999

    Google Scholar 

  5. Mehta NP, Kramer DB: A critique of the USMLE clinical skills examination. MedScape Gen Med 2005; 7: 76

    Google Scholar 

  6. History of SP bl]Reference, Howard Barrows and History of standardized patients, in History of Medicine

  7. Dupras DM, Li JT: Use of an objective structured clinical examination to determine clinical competence. Acad Med 1995; 70: 1029–1034

    Article  CAS  PubMed  Google Scholar 

  8. Rothman AI, Cusimano M: Assessment of English proficiency in international medical graduates by physician examiners and standardized patients. Med Educ 2001; 35: 762–766

    Article  CAS  PubMed  Google Scholar 

  9. Humphrey-Murto S, Smee S, Touchie C, et al: A comparison of physician examiners and trained assessors in a high-stakes OSCE setting. Acad Med 2005; 80: S59–62

    Article  PubMed  Google Scholar 

  10. Tamblyn RM, Klass DK, Schandl GK, et al: Factors associated with the accuracy of standardized patient presentation. Acad Med 1990; 65: S55–S56

    Article  CAS  PubMed  Google Scholar 

  11. Carney PA, Dietrich AJ, Freeman DH, et al: A standardized-patient assessment of a continuing medical education program to improve physicians’ cancer-control clinical skills. Acad Med 1995; 70: 52–58

    Article  CAS  PubMed  Google Scholar 

  12. Cohen DS, Colliver JA, Marcy MS, et al: Psychometric properties of a standardized-patient check list and rating form used to assess interpersonal and communications skills. Acad Med 1995; 71: S87–S89

    Article  Google Scholar 

  13. Schnable G, Hassard TH, Kopelow ML: The assessment of interpersonal skills using standardized patients. Acad Med 1991; 66: S34–S36

    Article  Google Scholar 

  14. Stillman P, Swanson D, Regan MB, et al: Assessment of clinical skills of residents utilizing standardized patients. Ann Intern Med 1991; 114: 393–401

    Article  CAS  PubMed  Google Scholar 

  15. McLeod PJ, Tamblyn RM, Gayton D, et al: Use of standardized patients to assess between: physician variations in resource utilization. JAMA 1997; 278: 1164–118

    Article  CAS  PubMed  Google Scholar 

  16. Regehr G, Freeman R, Hodges B, et al: Assessing the generalizability of OSCE measures across content domains. Acad Med 1999; 74: 1320–1322

    Article  CAS  PubMed  Google Scholar 

  17. VanMents M: The Effective Use of Role-Playing: A Handbook for Teachers and Trainers (revised ed.). Edited by Kogan P. New York, London/Nichols Publishing, 1989

  18. Kravitz RL, Epstein RM, Feldman MD, et al: Influence of patients’ requests for direct-to-consumer advertised antidepressants: a randomized controlled trial. JAMA 2005; 293: 1995–2002

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  19. Glassman PA, Luck J, O’Gara EM, et al: Using standardized patients to measure quality: evidence from the literature and a prospective study. J Qual Improv 2000; 26: 644–653

    CAS  Google Scholar 

  20. Hoppe RB, Farquhar LJ, Henry R, et al: Residents’ attitudes towards and skills in counseling: using undetected standardized patients. J Gen Intern Med 1990; 5: 415–420

    Article  CAS  PubMed  Google Scholar 

  21. Kennedy C, Yellowlees P: The effectiveness of telepsychiatry measured using the Health of the Nation Outcome Scale and the Mental Health Inventory. J Telemed Telecare 2003; 9: 12–16

    Article  PubMed  Google Scholar 

  22. Kapur PA, Steadman RH: Patient simulator competency testing: ready for takeoff? Anesth Analg 1998; 86: 1157–1159

    CAS  PubMed  Google Scholar 

  23. Morgan PJ, Cleave-Hogg D: A worldwide survey of the use of simulation in anesthesia. Can J Anesth 2002; 49: 659–662

    Article  PubMed  Google Scholar 

  24. Aggarwal R, Moorthy K, Darzi A: Laparoscopic skills training and assessment. Br J Surg 2004; 91: 1549–1558

    Article  CAS  PubMed  Google Scholar 

  25. Datta V, Bann S, Beard J, et al: Comparison of bench test evaluations of surgical skill with live operating performance assessments. J Am Coll Surg 2004; 199: 603–606

    Article  PubMed  Google Scholar 

  26. Korndorffer JR, Scott DJ, Sierra R, et al: Developing and testing competency levels for laparoscopic skills training. Arch Surg 2005; 140: 80–84

    Article  PubMed  Google Scholar 

  27. Byrne AJ, Greaves JD: Assessment instruments used during anesthetic simulation: review of published studies. Br J Anaesth 2001; 86: 445–450

    Article  CAS  PubMed  Google Scholar 

  28. Murray D, Boulet J, Ziv A, et al: An acute care skills evaluation for graduating medical students: a pilot study using clinical simulation. Med Educ 2002; 36: 833–841

    Article  PubMed  Google Scholar 

  29. Murray DJ, Boulet JR, Kras JF, et al: A simulation-based acute skills performance assessment for anesthesia training. Anesth Analg 2005; 101: 1127–1134

    Article  PubMed  Google Scholar 

  30. Clyne B, Gutman D, Sutton E, et al: Oral board versus high fidelity simulation for competency assessment: senior emergence medicine resident management of an acute coronary syndrome. Presented at International Meeting on Medical Simulation, Albuquerque, NM, 2004. Available at www.anestech.org/publications_abstracts.htm.

  31. Sudikoff SN: High fidelity medical simulation as an assessment tool for pediatric resident airway management skills. Poster presentation at International Meeting on Medical Simulation. Miami, Fla, 2005. Available at www.anestech.org/publications_abstracts.htm.

  32. Christensen J, Patel P, Skinner C: The use of patient simulators for residency competency evaluation. Poster presentation at International Meeting on Medical Simulation, San Diego, Calif, 2006. Available at www.anestech.org/publications_abstracts.htm.

  33. Rosenblatt MA, Abrams KJ, New York State Society of Anesthesiologists Committee on Continuing Medical Education and Remediation, Remediation Sub-Committee: The use of a human patient simulator in the evaluation of and development of a remedial prescription for an anesthesiologist with lapsed medical skills. Anesth Analg 2002; 94: 140–153

    Google Scholar 

  34. Berkenstadt H, Ziv A, Dach R, et al: The process of incorporating simulation-based competency assessment into the Israeli National Board Examination in Anesthesiology. Presented at International Meeting on Medical Simulation. Albuquerque, NM, 2004. Available at www.anestech.org/publications_abstracts.htm.

  35. Boulet JR, Swanson DB: Psychometric challenges of using simulations for high-stakes assessment, in Simulators in Critical Care and Beyond: The Society of Critical Care Medicine. Edited by Dunn WF. 2004, pp 119–130

  36. Hotchkiss MA, Biddle C, Fallacaro M: Assessing the authenticity of the human simulation experience in anesthesiology. AANA J 2002; 70: 470–473

    PubMed  Google Scholar 

  37. Najjar LJ: Principles of educational multimedia user interface design. Hum Factors 1998; 40: 311–324

    Article  Google Scholar 

  38. Inwood MJ, Ahmad J: Development of instructional, interactive, multimedia anatomy dissection software: a student-led initiative. Clin Anat 2005; 18: 613–617

    Article  PubMed  Google Scholar 

  39. Issenberg SB, Gordon MS, Greber AA: Bedside cardiology skills training for the osteopathic internist using simulation technology. J Am Osteopath Assoc 2003; 103: 603–607

    PubMed  Google Scholar 

  40. Seabra D, Srougi M, Baptista R, et al: Computer aided learning versus standard lecture for undergraduate education in urology. J Urol 2004; 171: 1220–1222

    Article  PubMed  Google Scholar 

  41. Williams C, Aubin S, Harkin P, et al: A randomized, controlled, single-blind trial of teaching provided by a computer-based multimedia package versus lecture. Med Educ 2001; 35: 847–854

    Article  CAS  PubMed  Google Scholar 

  42. Wiederhold BK: Virtual Healing. San Diego, Calif, Interactive Media Institute, 2004

    Google Scholar 

  43. Vichitvejpaisal P, Sitthikongsak S, Preechakoon B, et al: Does computer-assisted instruction really help to improve the learning process? Med Educ 2001; 35: 983–989

    CAS  PubMed  Google Scholar 

  44. Yellowlees P, MacKenzie J: Telehealth responses to bio-terrorism and emerging infections. J Telemed Telecare 2003; 9: S80–2

    Article  PubMed  Google Scholar 

  45. Humphris GM, Kaney S: The Objective Structured Video Exam for assessment of communication skills. Med Educ 2000; 34: 939–945

    Article  CAS  PubMed  Google Scholar 

  46. Hulsman RL, Mollema ED, Oort FJ, et al: Using standardized video cases for assessment of medical communication skills: reliability of an objective structured video examination by computer. Patient Educ Couns 2006; 60: 24–31

    Article  CAS  PubMed  Google Scholar 

  47. Parker P: The Courage to Teach: The Inner Landscape of a Teacher’s Life. San Francisco, Jossey-Bass, 1998

    Google Scholar 

  48. Pratt DD, Collins J: Five Perspectives on Teaching in Adult & Higher Education. Malabar, Fla, Krieger Publishing, 1998

    Google Scholar 

  49. Kember D: A reconceptualization of the research into university academics: conceptions of teaching. Learn Instruct 1997; 7: 255–275

    Article  Google Scholar 

  50. Mitchell M, Srinivasan M, Franks P, et al: Resident physician performance: an examination of state of the literature, and an integrative performance model. Acad Med. 2005

  51. Srinivasan M, Litzelman D, Seshadri R, et al: Confronting unprofessional learner behavior: development of an instrument to code educator responses. Acad Med. 2004. Sept.

  52. Crocker L, Algina J: Introduction of Classical and Modern Test Theory. CBS College Publishing, 1986

  53. Cronbach LJ: Essentials of Psychological Testing. Harper & Row Publishers, 970

  54. Streiner DL, Norman GR: Health Measurement Scales: A Practical Guide to Their Development and Use. Oxford Medical Publications.

  55. Shavelson RJ, Webb NM: Generalizability Theory: A Primer. Newbury Park, Calif, Sage Publications, 1991, pp 1–16

    Google Scholar 

  56. Shavelson RJ, Webb NM, Rowley GL: Generalizabiltytheory. Am Psychol 1989; 44: 922–932

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Medicine, University of California, Davis, Sacramento, California, USA

    Malathi Srinivasan M.D.,  Judith C. Hwang M.D., M.B.A.,  Daniel West M.D. &  Peter M. Yellowlees M.B.B.S., M.D.

Authors
  1. Malathi Srinivasan M.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Judith C. Hwang M.D., M.B.A.
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniel West M.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Peter M. Yellowlees M.B.B.S., M.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Malathi Srinivasan M.D..

Rights and permissions

Reprints and permissions

About this article

Cite this article

Srinivasan, M., Hwang, J.C., West, D. et al. Assessment of Clinical Skills Using Simulator Technologies. Acad Psychiatry 30, 505–515 (2006). https://doi.org/10.1176/appi.ap.30.6.505

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1176/appi.ap.30.6.505

Keywords

Search

Navigation