Advertisement

The reliability of a portfolio of workplace-based assessments in anesthesia training

  • Damian J. CastanelliEmail author
  • Joyce M. W. Moonen-van Loon
  • Brian Jolly
  • Jennifer M. Weller
Reports of Original Investigations

Abstract

Purpose

Competency-based anesthesia training programs require robust assessment of trainee performance and commonly combine different types of workplace-based assessment (WBA) covering multiple facets of practice. This study measured the reliability of WBAs in a large existing database and explored how they could be combined to optimize reliability for assessment decisions.

Methods

We used generalizability theory to measure the composite reliability of four different types of WBAs used by the Australian and New Zealand College of Anaesthetists: mini-Clinical Evaluation Exercise (mini-CEX), direct observation of procedural skills (DOPS), case-based discussion (CbD), and multi-source feedback (MSF). We then modified the number and weighting of WBA combinations to optimize reliability with fewer assessments.

Results

We analyzed 67,405 assessments from 1,837 trainees and 4,145 assessors. We assumed acceptable reliability for interim (intermediate stakes) and final (high stakes) decisions of 0.7 and 0.8, respectively. Depending on the combination of WBA types, 12 assessments allowed the 0.7 threshold to be reached where one assessment of any type has the same weighting, while 20 were required for reliability to reach 0.8. If the weighting of the assessments is optimized, acceptable reliability for interim and final decisions is possible with nine (e.g., two DOPS, three CbD, two mini-CEX, two MSF) and 15 (e.g., two DOPS, eight CbD, three mini-CEX, two MSF) assessments respectively.

Conclusions

Reliability is an important factor to consider when designing assessments, and measuring composite reliability can allow the selection of a WBA portfolio with adequate reliability to provide evidence for defensible decisions on trainee progression.

Fiabilité d’un portfolio d’évaluations sur le lieu de travail dans la formation en anesthésie

Résumé

Objectif

Les programmes de formation en anesthésie basés sur les compétences nécessitent de solides évaluations des performances des stagiaires et combinent habituellement des évaluations sur le lieu de travail (ÉLT) couvrant de nombreux aspects de la pratique. Cette étude a mesuré la fiabilité des ÉLT dans une grande base de données existante et a exploré comment elles pourraient être combinées pour accroître leur fiabilité pour des décisions sur les évaluations.

Méthodes

Nous avons utilisé la théorie de la généralisation pour mesurer un critère composite de fiabilité de quatre types d’ÉLT utilisés par les collèges d’anesthésiologistes d’Australie et de Nouvelle-Zélande : un exercice de mini-évaluation clinique (mini-CEX), l’observation directe des habiletés procédurales (DOPS), une discussion de cas (CbD) et une rétroaction de multiples sources (MSF). Nous avons alors modifié le nombre et la pondération des combinaisons d’ÉLT pour optimiser la fiabilité avec moins d’évaluations.

Résultats

Nous avons analysé 67 405 évaluations de 1 837 stagiaires et 4 145 assesseurs. Nous avons supposé une fiabilité acceptable pour les décisions intérimaires (enjeux intermédiaires) et définitives (enjeux élevés) à, respectivement, 0,7 et 0,8. Selon la combinaison des types d’ÉLT, 12 évaluations ont permis d’atteindre le seuil de 0,7 lorsqu’une évaluation de chaque type a le même poids, alors qu’il en a fallu 20 pour que la fiabilité atteigne 0,8. Si la pondération des évaluations est optimisée, la fiabilité acceptable pour les décisions intérimaires et finales est possible avec, respectivement, neuf évaluations (p. ex., deux DOPS, trois CbD, deux mini-CEX, deux MSF) et quinze évaluations (p. ex. deux DOPS, huit CbD, trois mini-CEX, deux MSF).

Conclusions

La fiabilité est un facteur important dont il faut tenir compte quand on conçoit les évaluations et la mesure d’une fiabilité composite permet la sélection d’un éventail d’ÉLT avec une fiabilité adéquate pour l’obtention de données probantes et la défense de décisions sur les progrès des stagiaires.

Notes

Conflicts of interest

None declared.

Editorial responsibility

This submission was handled by Dr. Gregory L. Bryson, Deputy Editor-in-Chief, Canadian Journal of Anesthesia.

Author contributions

Damian J. Castanelli designed the study, contributed to data management, analysis, and interpretation, and wrote the manuscript. Joyce M.W. Moonen-van Loon contributed to study design, led the analysis, contributed to interpretation, and commented on each draft of the manuscript. Brian Jolly contributed to study design, analysis and interpretation, drafting the manuscript, and commented on each draft of the manuscript. Jennifer M. Weller contributed to study design, interpretation, drafting the manuscript, and commented on each draft of the manuscript.

Funding

This work was supported by a grant from the Australian and New Zealand College of Anaesthetists, (ANZCA grant number: S16/043).

References

  1. 1.
    Australian and New Zealand College of Anaesthetists. Anaesthesia training program curriculum. Melbourne: Australian and New Zealand College of Anaesthetists, 2012. Available from URL: http://www.anzca.edu.au/documents/anaesthesia-training-program-curriculum.pdf (accessed October 2018).
  2. 2.
    Van Gessel E, Mellin-Olsen J, Ostergaard HT. Niemi-Murola L; Education and Training Standing Committee, European Board of Anaesthesiology, Reanimation and Intensive Care. Postgraduate training in anaesthesiology, pain and intensive care: the new European competence-based guidelines. Eur J Anaesthesiol 2012; 29: 165-8.CrossRefGoogle Scholar
  3. 3.
    Stodel EJ, Wyand A, Crooks S, Moffett S, Chiu M, Hudson CC. Designing and implementing a competency-based training program for anesthesiology residents at the University of Ottawa. Anesthesiol Res Pract 2015; 2015: 713038.Google Scholar
  4. 4.
    van der Vleuten CP, Schuwirth LW, Driessen EW, et al. A model for programmatic assessment fit for purpose. Med Teach 2012; 34: 205-14.CrossRefGoogle Scholar
  5. 5.
    Bok HG, Teunissen PW, Favier RP, et al. Programmatic assessment of competency-based workplace learning: when theory meets practice. BMC Med Educ 2013; 13: 123.CrossRefGoogle Scholar
  6. 6.
    Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007; 29: 855-71.CrossRefGoogle Scholar
  7. 7.
    Weller JM, Castanelli DJ, Chen Y, Jolly B. Making robust assessments of specialist trainees’ workplace performance. Br J Anaesth 2017; 118: 207-14.CrossRefGoogle Scholar
  8. 8.
    Weller JM, Jolly B, Misur MP, et al. Mini-clinical evaluation exercise in anaesthesia training. Br J Anaesth 2009; 102: 633-41.CrossRefGoogle Scholar
  9. 9.
    Weller JM, Misur M, Nicolson S, et al. Can I leave the theatre? A key to more reliable workplace-based assessment. Br J Anaesth 2014; 112: 1083-91.CrossRefGoogle Scholar
  10. 10.
    Watson MJ, Wong DM, Kluger R, et al. Psychometric evaluation of a direct observation of procedural skills assessment tool for ultrasound-guided regional anaesthesia. Anaesthesia 2014; 69: 604-12.CrossRefGoogle Scholar
  11. 11.
    Moonen-van Loon JM, Overeem K, Donkers HH, van der Vleuten CP, Driessen EW. Composite reliability of a workplace-based assessment toolbox for postgraduate medical education. Adv Health Sci Educ Theory Pract 2013; 18: 1087-102.CrossRefGoogle Scholar
  12. 12.
    Wragg A, Wade W, Fuller G, Cowan G, Mills P. Assessing the performance of specialist registrars. Clin Med (Lond) 2003; 3: 131-4.CrossRefGoogle Scholar
  13. 13.
    Downing SM. Reliability: on the reproducibility of assessment data. Med Educ 2004; 38: 1006-12.CrossRefGoogle Scholar
  14. 14.
    Crossley J, Russell J, Jolly B, et al. ‘I’m pickin’ up good regressions’: the governance of generalisability analyses. Med Educ 2007; 41: 926-34.CrossRefGoogle Scholar
  15. 15.
    Moonen-van Loon JM, Overeem K, Govaerts MJ, Verhoeven BH, van der Vleuten CP, Driessen EW. The reliability of multisource feedback in competency-based assessment programs: the effects of multiple occasions and assessor groups. Acad Med 2015; 90: 1093-9.CrossRefGoogle Scholar
  16. 16.
    Crossley J, Jolly B. Making sense of work-based assessment: ask the right questions, in the right way, about the right things, of the right people. Med Educ 2012; 46: 28-37.CrossRefGoogle Scholar
  17. 17.
    Pelgrim EA, Kramer AW, Mokkink HG, van den Elsen L, Grol RP, van der Vleuten CP. In-training assessment using direct observation of single-patient encounters: a literature review. Adv Health Sci Educ Theory Pract 2011; 16: 131-42.CrossRefGoogle Scholar
  18. 18.
    Lockyer J, Violato C, Fidler H. A multi source feedback program for anesthesiologists. Can J Anesth 2006; 53: 33-9.CrossRefGoogle Scholar
  19. 19.
    Brennan RL. Generalizability Theory and classical test theory. J Appl Meas Educ 2010; 24: 1-21.CrossRefGoogle Scholar
  20. 20.
    van der Vleuten CP, Schuwirth LW, Scheele F, Driessen EW, Hodges B. The assessment of professional competence: building blocks for theory development. Best Pract Res Clin Obstet Gynaecol 2010; 24: 703-19.CrossRefGoogle Scholar
  21. 21.
    Castanelli DJ, Jowsey T, Chen Y, Weller JM. Perceptions of purpose, value, and process of the mini-Clinical Evaluation Exercise in anesthesia training. Can J Anesth 2016; 63: 1345-56.CrossRefGoogle Scholar
  22. 22.
    Driessen EW, van Tartwijk J, Govaerts M, Teunissen P, van der Vleuten CP. The use of programmatic assessment in the clinical workplace: a Maastricht case report. Med Teach 2012; 34: 226-31.CrossRefGoogle Scholar
  23. 23.
    Van Tartwijk J, Driessen EW. Portfolios for assessment and learning: AMEE Guide no. 45. Med Teach 2009; 31: 790-801.CrossRefGoogle Scholar
  24. 24.
    Ten Cate O, Chen HC, Hoff RG, Peters H, Bok H, van der Schaaf M. Curriculum development for the workplace using Entrustable Professional Activities (EPAs): AMEE Guide No. 99. Med Teach 2015; 37: 983-1002.CrossRefGoogle Scholar
  25. 25.
    Wisman-Zwarter N, van der Schaaf M, Ten Cate O, Jonker G, van Klei WA, Hoff RG. Transforming the learning outcomes of anaesthesiology training into entrustable professional activities: a Delphi study. Eur J Anaesthesiol 2016; 33: 559-67.CrossRefGoogle Scholar
  26. 26.
    Jonker G, Hoff RG, Ten Cate OT. A case for competency-based anaesthesiology training with entrustable professional activities: an agenda for development and research. Eur J Anaesthesiol 2015; 32: 71-6.CrossRefGoogle Scholar
  27. 27.
    Eva KW, Macala C. Multiple mini-interview test characteristics: ‘tis better to ask candidates to recall than to imagine. Med Educ 2014; 48: 604-13.CrossRefGoogle Scholar

Copyright information

© Canadian Anesthesiologists' Society 2018

Authors and Affiliations

  1. 1.School of Clinical Sciences at Monash HealthMonash UniversityClaytonAustralia
  2. 2.Department of Anaesthesia and Perioperative MedicineMonash HealthClaytonAustralia
  3. 3.Department of Educational Development and Research, Faculty of Health, Medicine, and Life SciencesMaastricht UniversityMaastrichtThe Netherlands
  4. 4.School of Medicine and Public Health, Faculty of Health and MedicineUniversity of NewcastleNewcastleAustralia
  5. 5.Centre for Medical and Health Sciences Education, School of MedicineUniversity of AucklandAucklandNew Zealand
  6. 6.Department of AnaesthesiaAuckland City HospitalAucklandNew Zealand

Personalised recommendations