Abstract
Introduction
One of the most challenging aspects of Emergency Medicine (EM) residency is mastering the leadership skills required during a resuscitation. Use of resuscitation video recording for debriefing is gaining popularity in graduate medical education. However, there are limited studies of how video technology can be used to improve leadership skills in the emergency department. We aim to evaluate the utility of video-assisted self-reflection, compared with self-reflection alone, in the setting of resuscitation leadership.
Methods
This was a prospective, randomized, controlled pilot study conducted in 2018 at an urban level 1 trauma center with a three-year EM residency program. The trial included postgraduate year (PGY) 2 and 3 residents (n = 10). Each resident acted as an individual team leader for a live real-time resuscitation in the emergency department. The authors classified a patient as a resuscitation if there was an immediate life- or limb-threatening disease process or an abnormal vital sign with an indication of hypoperfusion. Each resident was recorded as the team leader twice. Both control and intervention groups produced written self-reflection after their first recording. The intervention group viewed their resuscitation recording while completing the written reflection. After their reflection, all participants were recorded for a second resuscitation. Two faculty experts, blinded to the study, scored each video using the Concise Assessment of Leader Management (CALM) scale to measure the leadership skills of the resident team leader.
Results
Five PGY‑3 and five PGY‑2 residents participated. The weighted kappa between the two experts was 0.45 (CI 0.34–0.56, p < 0.0001). The median gain score in the control group was −1.5 (IQR) versus 0.5 in the intervention group (IQR).
Discussion
Video-assisted self-reflection showed positive gain score trends in leadership evaluation for residents during a resuscitation compared with the non-video assisted control group. This tool would be beneficial to implement in EM residency.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Introduction
Leadership failure can have disastrous results. A report by The Joint Commission found that team leadership failures in acute healthcare settings were associated with more than 50% of adverse events [1]. Exacerbating this is that medical education has been slow to incorporate explicit leadership training into graduate medical education [2, 3]. Many residents develop leadership skills during patient care through trial and error, observation, feedback, and reflection [4]. Specific to Emergency Medicine (EM), the modality by which team leadership is effectively assessed has been elusive [4, 5].
One of the most challenging aspects of EM residency training is mastering the leadership skills required during a resuscitation[4]. During a resuscitation, the team leader is tasked with the responsibility of gathering information from multiple sources, formulating a differential diagnosis, initiating interventions, and communicating clearly and professionally with the patient and team members. These tasks must be performed quickly and competently in critically ill patients. There are critical moments where the leader’s actions, communication and decisions have a large impact on the patient. This is usually an intense period during which leaders can be cognitively overloaded, making it inappropriate or impossible for them to reflect on their leadership abilities in real time [6].
Simulation allows for complete control of the resuscitation case to practice logistics, understand procedures, and demonstrate learning points. However, the disadvantage of simulated cases is that they lack the authenticity, pressure and investment that occurs when treating a real, live unstable patient [7, 8].
Debriefing and self-reflection are ways to address the strengths and weaknesses of leadership skills. This practice of analyzing decisions that unfold quickly and have a high impact is commonly used in elite athletics. The exercise takes advantage of behaviorism and visualization to understand what they did, why they did it, and how they will change their behavior next time for a better outcome. Additionally, athletic reflection is often video-assisted. It is seen in the stereotypical situation of football players watching films before a game and has been studied in volleyball and tennis, showing improved long-term memory in tactical knowledge, improved strategy planning, and more meaningful awareness of the environment [9, 10].
There is a tremendous opportunity to translate video-assisted self-reflection to leadership skills in resuscitations. Video-assisted feedback has been shown to improve algorithmic compliance in Advanced Trauma Life Support (ATLS) and cardiopulmonary resuscitation (CPR), with the premise that objective evidence of performance improves motivation and self-efficacy [11, 12].
The basis for the hypothesis that skills will improve more amongst residents who are able to watch videos of themselves leading a resuscitation has its roots in cognitivism. Reflection is critical in cognitivism, the first step of which is to return to and replay the learner’s experience [13]. This step is facilitated by providing learners with a video of themselves as team leader so that the learner can replay, with 100% accuracy, the events of the resuscitation. In cognitivism, the onus of learning is firmly on the learner, with the teacher acting as a guide to help the learner “learn how to learn [13]”.
In this study, we assess the utility of resuscitation video-assisted self-reflection compared with self-reflection alone.
Methods
We conducted a prospective, randomized, controlled pilot study. Audio and video were recorded of EM residents functioning as team leader during a resuscitation in real-time patients presenting to the emergency department. Recordings were uploaded to an encrypted web-based storage system (Wayne State One Drive) accessible only to the study authors. The Wayne State University School of Medicine Institutional Review Board classified this study as program evaluation/quality improvement and granted exemption. Patient consent was not required based on this classification as quality improvement. We have also adhered to the CONSORT guideline [14].
Setting and population
This study was conducted at an urban, level 1 trauma center with approximately 90,000 adult visits per year and a three-year EM residency program. A total of ten PGY‑2 and PGY‑3 EM resident physicians volunteered to enroll in the study. Recruitment consisted of an email sent to the second- and third-year classes by a PGY‑3 participating in the study offering the opportunity to participate. Study staff would then be present in the emergency department during their shifts to capture data. Inclusion criteria for resident participation included (1) second- or third-year EM residents in good standing, (2) on an EM rotation during the months of August, September, and October 2018, and (3) having a resuscitation on a day that a member of the study staff was present to record. Patients were included if they were triaged as a resuscitation by emergency medical services or a triage nurse, which comprises patients who have an immediate life- or limb-threatening disease process, or patients with a vital sign abnormality with clear evidence of hypoperfusion.
Study protocol
The ten resident physicians were randomized into two groups, with an allocation ratio of 1:1. The control group consisted of three PGY‑2 residents and two PGY‑3 residents. The intervention group consisted of two PGY‑2 residents and three PGY‑3 residents. Participants in both groups were recorded leading an adult resuscitation twice for a total of 20 recordings. After the first resuscitation, all ten residents participated in guided self-reflection regarding their capabilities as team leader. For all ten residents, the self-reflection was guided in that they were provided with a “Resident Reflection Sheet” (Supplemental File 1 of the Electronic Supplemental Material) as opposed to reflecting on their capabilities in an unstructured way. The intervention group watched a recording of their resuscitation while they completed the “Resident Reflection Sheet,” whereas the control group completed the “Resident Reflection Sheet” without access to their recording. All residents in each group were then recorded leading a second resuscitation. No feedback or discussion regarding the resident’s self-reflection was provided prior to completion of the second resuscitation.
Two faculty experts with experience in residency leadership and education reviewed all 20 videos to assess the leadership abilities of the resident functioning as the team leader during the case. The faculty experts reviewed the videos in random order and did not know whether they were viewing a video from the control or the intervention group, nor were they aware whether they were viewing a video from the first group of resuscitation cases (prior to the guided self-reflection) or after the guided self-reflection. The faculty experts scored all videos using the Concise Assessment of Leader Management (CALM) instrument (Supplemental File 2 of the Electronic Supplementary Material), a previously validated instrument designed to require minimal user training to assess leadership skills and to provide formative feedback to learners [15]. Rater training of the faculty experts consisted of a consensus meeting to agree on interpretation of assessment variables, and three practice cases wherein the experts rated cases independently and discussed differences in scoring between the two. These practice cases were not included in the overall cohort. Residents were unaware that their performance would be evaluated by the CALM instrument.
The key outcome measure in this study was the CALM score before and after video-assisted reflection.
Data analysis
Descriptive statistics were used to explore resident characteristics and evaluation timing; ordinal data were described using medians and interquartile ranges (IQRs). Individual scores for each resident generated using the CALM instrument for the pre- and post-test assessments were summed for a total score. A gain score was calculated for each resident by subtracting pre-test scores from post-test scores. Total gain scores on the CALM instrument between the control and intervention groups were compared. Gain scores between the control and intervention group for the CALM instrument were reported as medians (IQR) with a 95% confidence interval (CI). Agreement between the two faculty experts was assessed using the weighted kappa statistic. Consensus was also assessed between the two faculty experts, and was defined as a variation of no greater than one point on the CALM score for each of the 15 individual items in the instrument to account for subjectivity in ratings while still allowing for identification of significant differences in ratings.
Results
Five PGY‑2 residents and five PGY‑3 residents participated in the study out of 28 possible candidates. All video recording took place in September and October 2018. The median time between the first resuscitation and resident reflection was three days (mean 9.2 days). The median time between the first and second resuscitation recordings was 16.5 days (mean 19.3 days). Resuscitation cases encountered during the study period are described in Tab. 1.
Consensus was achieved between the two faculty experts for 95% of the 300 total variables on the CALM assessment tool (15 variables for each video times 20 videos). The weighted Kappa between the two experts was 0.45 (CI 0.34–0.56, p < 0.0001).
Tab. 2 shows the total median composite gain scores of control vs. intervention, the median gain scores generated by the expert reviewers between the residents’ first and second recordings, and also describes all 15 items assessed with the CALM instrument. The median composite gain score in the control group was −1.5 (IQR) versus 0.5 in the intervention group (IQR). The 95% CI for the difference in the medians between the groups was −8.5 to 4.0. Gain scores for the individual faculty experts were also compared and no significant difference was found for either the control or intervention group (data not shown).
Discussion
Functioning effectively as the team leader during a resuscitation is a difficult task to master [12, 16]. Resident learners may not be able to identify their shortcomings as team leader given that physicians have been found to be poor at self-assessment [17]. Prior work has shown that video analysis leads to improved algorithmic compliance in trauma resuscitation cases and in cardiopulmonary resuscitation [11, 12, 18, 19]. We have applied this line of thinking to leadership in resuscitations, where there is not a specific algorithm to comply with, to test whether residents watching videos of themselves leading a resuscitation will gain the same benefit conferred to physicians who are able to watch themselves following the ATLS or CPR algorithm.
This pilot study found a positive trend in gain scores using the CALM instrument and demonstrates a positive effect of incorporating video feedback into leadership training for EM residents. Previous studies have examined the relationship between resident self-assessment and faculty assessment of resident performance when watching videos of residents performing elective laparoscopic surgery and in placing ultrasound-guided internal jugular triple lumen catheters and found no correlation between the residents’ self-assessment scores and the scores given by faculty after watching the videos [20, 21]. This study differs in that we were not looking for a correlation between these two scores but rather for an improvement in objective faculty assessment of resident performance if the residents were provided with their videos to watch during their guided self-assessment.
Given aspects of the CALM instrument such as prioritizing task order or stating next steps in patient care are likely obvious to learners when watching themselves, it may not be surprising that gain scores showed improvement in the intervention group. However, it is important to note that the resident had no knowledge of being assessed by the CALM instrument and therefore did not know which specific domains were to be judged. Clearly for the purpose of this study, we could not provide the resident with the CALM assessment tool in order to not confound our results, but we expect the effects of video feedback to be significantly magnified once the residents are aware of its purpose. Additionally, residents can be trained to appraise their videos critically so that they can obtain maximum benefit from watching themselves and can target many different skills, in addition to leadership.
This project’s assessment of leadership skills provides benefit of data for several competencies of the Emergency Medicine Milestones Project [22]. The Accreditation Council of Graduate Medical Education (ACGME) and the American Board of Emergency Medicine created the Emergency Medicine Milestones Project in 2012 to provide competency-based assessment recommendations for EM residents in ACGME-approved residencies. Specifically, the Patient Safety competency (SBP1) requires that an EM resident “identifies situations when the breakdown in teamwork or communication may contribute to medical error” in order to achieve a level four milestone. The Practice-based Performance Improvement competency (PBL1) requires that a resident “performs self-assessment to identify areas for continued self-improvement” to achieve a level three milestone. Finally, the Team Management competency (ICS2) generally requires that a resident “leads patient-centered care teams, ensuring effective communication and mutual respect among members of the team” in order to achieve any level of this milestone and thus is the competency most well addressed with video analysis. All of these milestones can be objectively assessed with the methods employed in this study and may be of value to residency programs wishing to identify further objective data for these competencies [22].
Another potential usage of video recording for feedback in EM is in the realm of directed coaching. This has already been studied in general surgery, wherein resident physicians were video recorded performing a surgical procedure. Each resident then met with a supervising attending physician for 1 hour and were provided with targeted feedback while watching the video together. This resulted in improved and individualized teaching and feedback [23]. Soucisse et al. found improved surgical skills in general surgery residents filmed performing side-by-side intestinal anastomoses who were then provided coaching by faculty while reviewing the video together [24]. The usage of video recordings with debriefing and coaching by faculty represents a future direction for using video recording in EM residency training that merits exploration. Additional recording modalities, such as usage of a head-mounted GoPro, have been studied in general surgery residents with results demonstrating feasibility of this modality and utility in providing more meaningful feedback [25]. Use of an inexpensive, easy-to-acquire recording device such as a GoPro may improve the feasibility of video recording and feedback in EM.
This study begins to shed light on the difficulties of effective leadership development during resuscitations and the benefit of reflection with objective video recording. Our goal would be to incorporate this exercise in residency education. Other than the initial cost of the recording equipment, this is not a practice that requires continuous funding or significant faculty time or oversight. Additionally, residents can perform this activity on their own and present their findings to faculty or program leadership, which is a major advantage over many other curricular innovations.
Limitations
The primary limitations are the small sample size and single site nature of this pilot study. The study participants were also drawn from a convenience sampling of EM residents, which may confound the results. There was also some variability in the timing of when participants completed their self-reflection relative to completion of the resuscitation case, which may affect the influence of the self-reflection exercise. Given that the resuscitations were on live patients presenting to the emergency department rather than simulated cases, there was heterogeneity in the complexity of the case presentations which could affect the resident’s performance. Additionally, reflection was done solely by the resident and not with the whole medical team (nursing staff, respiratory therapist, pharmacy staff), which may give further insight into the residents leadership abilities.
Conclusions
Our study showed a positive trend in gain score evaluation of leadership skills for residents utilizing video-assisted self-reflection after resuscitation compared with the non-video-assisted control group. These initial results are very encouraging and support further exploration with a larger sample size and multiple sites.
References
No authors listed. Update: sentinel event statistics. Jt Comm Perspect. 2006;26:14–5.
Kuo AK, Thyne SM, Chen HC, West DC, Kamei RK. An innovative residency program designed to develop leaders to improve the health of children. Acad Med. 2010;85:1603–8.
Hartzell JD, Yu CE, Cohee BM, Nelson MR, Nelson MR, Wilson RL. Moving beyond accidental leadership: a graduate medical education leadership curriculum needs assessment. Mil Med. 2017;182:e1815–e22.
Goldman EF, Plack MM, Roche CN, Smith JP, Turley CL. Leaning clinical versus leadership competencies in the emergency department: strategies, challenges, and supports of emergency medicine residents. J Grad Med Educ. 2011;3:320–5.
Rosenman ED, Branzetti JB, Fernandez R. Assessing team leadership in emergency medicine: the milestones and beyond. J Grad Med Educ. 2016;8:332–40.
Fraser KL, Ayres P, Sweller J. Cognitive load theory for the design of medical simulation. Simul Healthc. 2015;10:295–307.
Morgan P, Tregunno D, Brydges R, et al. Using a situational awareness global assessment technique for interprofessional obstetrical team training with high fidelity simulation. J Interprof Care. 2015;29:13–9.
Wildman JL, Salas E, Scott CP. Measuring cognition in teams: a cross-domain review. Hum Factors. 2014;56:911–41.
Garcia-Gonzalez L, Moreno MP, Moreno A, Gil A, Del Villar F. Effectiveness of a video-feedback and questioning programme to develop cognitive expertise in sport. Plos One. 2013;8:e82270.
Moreno MP, Moreno A, Garcia-Gonzalez L, Urena A, Hernandez C, Del Villar F. Intervention based on video feedback and questioning to improve tactical knowledge in expert female volleyball players. Percept Mot Skills. 2016;122:911–32.
Scherer L, Chang M, Wayne M, Battistella FD. Videotape review leads to rapid and sustained learning. Am J Surg. 2003;185:516–20.
Torre D, Daley B, Sebastian J, Elnicki DM. Overview of current learning theories for medical educators. Am Med J. 2006;119:903–7.
Lingard L, Vanstone M, Durrant M, et al. Conflicting messages: examining the dynamics of leadership on interprofessional teams. Acad Med. 2012;87:1762–7.
Eldridge SM, Chan CL, Campbell MJ, et al. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. Pilot Feasibility Stud. 2016;2:64.
Nadkarni LD, Roskind CG, Auerbach MA, Calhoun AW, Adler MD, Kessler DO. The development and validation of a concise instrument for formative assessment of team leader performance during simulated pediatric resuscitations. Simul Healthc. 2018;13:77–82.
Marsch S, Muller C, Marquardt K, Conrad G, Tschan F, Hunziker PR. Human factors affect the quality of cardiopulmonary resuscitation in simulated cardiac arrests. Resuscitation. 2004;60:51–6.
Davis D, Mazmanian P, Fordis M, Harrison RV, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence—A systematic review. JAMA. 2006;296:1094–102.
Wurster LA, Thakkar RK, Haley KJ, et al. Standardizing the initial resuscitation of the trauma patient with the Primary Assessment Completion Tool using video review. J Trauma Acute Care Surg. 2017;82:1002–6.
Ritchie PD, Cameron PA. An evaluation of trauma team leader performance by video recording. Aust N Z J Surg. 1999;69:183–6.
Herrera-Almario GE, Kirk K, Guerrero VT, Jeong K, Kim S, Hamad GG. The effect of video review of resident laparoscopic surgical skills measured by self- and external assessment. Am J Surg. 2016;211:315–20.
Wittler M, Hartman N, Manthey D, Hiestand B, Askew K. Video-augmented feedback for procedural performance. Med Teach. 2016;38:607–12.
The Emergency Medicine Milestone Project: A joint initiative of the Accreditation Council for Graduate Medical Education and the American Board of Emergency Medicine. 2015. http://www.acgme.org/acgmeweb/Portals/0/PDFs/Milestones/EmergencyMedicineMilestones.pdf. Accessed 3 June 2019.
Hu YY, Mazer LM, Yule SJ, et al. Complementing operating room teaching with video-based coaching. JAMA Surg. 2017;152:318–25.
Soucisse ML, Boulva K, Sideris L, Drolet P, Morin M, Dube P. Video coaching as an efficient teaching method for surgical residents—A randomized controlled trial. J Surg Educ. 2017;74:365–71.
Moore MD, Abelson JS, O’Mahoney P, Bagautdinov I, Yeo H, Watkins AC. Using GoPro to give video-assisted operative feedback for surgery residents: a feasibility and utility assessment. J Surg Educ. 2018;75:497–502.
Acknowledgements
We would like to acknowledge William Berk, MD, for his tremendous support in implementing this tool, and the emergency medicine residents and staff at the Emergency Department of Detroit Receiving Hospital for their participation.
Funding
The authors report no external funding source for.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
L. Kava, K. Jones, R. Ehrman, L. Smylie, M. McRae, E. Dubey, B. Reed and A. Messman declare that they have no competing interests.
Additional information
Prior poster presentation: Society for Academic Emergency Medicine Annual Scientific Assembly, Las Vegas, 2019, Gallery of Excellence Award.
Supplementary Information
40037_2021_690_MOESM1_ESM.pdf
1. Resident Reflection Sheet: used by each resident to guide self reflection after each resuscitation. The control group only used the resident reflection sheet. The intervention group filled out the resident reflection sheet while watching their video.
40037_2021_690_MOESM2_ESM.pdf
2. Concise Assessment of Leader Management: Previously validated instrument designed by Nadkarni LD, et. al to require minimal user training to assess leadership skills and provide formative feedback. Used by our attending experts to score all 20 videos.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Kava, L., Jones, K., Ehrman, R. et al. Video-assisted self-reflection of resuscitations for resident education and improvement of leadership skills: A pilot study. Perspect Med Educ 11, 80–85 (2022). https://doi.org/10.1007/s40037-021-00690-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40037-021-00690-9