Point-of-care ultrasound (POCUS) facilitates many diagnostic, procedural, and resuscitative applications in anesthesiology, including transesophageal echo (TEE) for cardiac and noncardiac surgery; transthoracic echocardiography for assessment of perioperative hemodynamic instability and volume status; lung ultrasound for identification of pneumothoraces, pleural effusions, and interstitial fluid; and procedural ultrasound for guiding vascular access and placement of neuraxial and peripheral nerve blocks.1,2,3,4,5,6 The use of POCUS has been shown to alter patient management decisions in the perioperative setting,1,7,8,9 and is important for clinical decision-making and patient outcomes.10 Nevertheless, evaluation of POCUS integration into anesthesiology education and practice is limited.3,8,11,12

Point-of-care ultrasound skills can be readily learned by anesthesiology trainees at all levels.1 Demonstrating the required POCUS skills to answer a clinical question is now an entrustable professional activity (EPA) of the Royal College of Physicians and Surgeons of Canada Anesthesiology training,13 and has been a required component of anesthesiology training in the UK since 2011.2 All Canadian anesthesiology programs currently provide some POCUS training in vascular access, peripheral nerve blocks, and transthoracic echocardiography, but the methods whereby training and competence are assessed and achieved vary considerably among programs.3 Point-of-care ultrasound curricula for anesthesiology trainees have been shown to improve learner satisfaction, exam scores, and clinical management.14 Point-of-care ultrasound training can improve short-term learning outcomes, but long-term retention of knowledge and skills may deteriorate without ongoing clinical application.8

Safe clinical use of POCUS requires proficiency in all aspects of ultrasound including knowledge, image acquisition, image interpretation, and integration of findings into clinical management.10,15 Nevertheless, there is a vast range of POCUS knowledge and skills among graduating medical students16 in different POCUS applications. Each POCUS application has a different degree of difficulty and consequent learning curve.

Recently, training and assessment of anesthesiology residents has transitioned to competency-based medical education (CBME).6,10,15,17,18 Competency-based medical education is an outcome-based approach for identifying desired skills and levels of performance along a continuum leading to independent practice and mastery. Despite recent plans to investigate the development of competency in POCUS applications for anesthesiology residents14,18 and a single study of the progression towards competence with lung ultrasound,19 no studies have determined how many supervised scans are required to achieve competence. Furthermore, there is no literature to inform whether anesthesiology residents can achieve competence in POCUS applications through participation in a structured anesthesiology POCUS curriculum, or whether POCUS exam skills in anesthesiology residents can be retained after a ≥ six-month hiatus.

A longitudinal anesthesia resident POCUS curriculum with flipped classroom video lectures,20 weekly live model scanning sessions, video review, and portfolio development was developed and implemented at the University of Saskatchewan in 2019. The flipped classroom is a learner-centred approach well suited to POCUS training, in which the direct instruction is undertaken by the learner through assigned online lectures. Instructor-guided small group live-scanning sessions are more interactive, with flexible hands-on learning opportunities.21 A six-month pause in the curriculum because of COVID-19 restrictions provided an opportunity to evaluate the retention of POCUS competency when the curriculum and study resumed.

We posed three primary research questions:

  1. 1.

    Can anesthesiology residents participating in a longitudinal POCUS curriculum achieve competence during residency?

  2. 2.

    How many supervised scans and how many sessions are needed to achieve a mean entrustment score ≥ 4 (“may use independently”) for each POCUS application in our curriculum?

  3. 3.

    Are POCUS competencies retained after a ≥ six-month pause or completion of curriculum?

Methods

Study design

Following ethics exemption as a program evaluation activity from the institutional Research Ethics Board (17 December 2018), and following approval (US-2168) from the local pandemic response and recovery team to resume increased activity on campus (University of Saskatchewan, Saskatoon, SK, Canada; September 2020), we conducted a prospective observational study to define the learning curve of anesthesiology residents participating in a longitudinal POCUS curriculum.

Participants

All anesthesiology residents in the foundations and core stages of training (PGY2–4) from January 2019 to March 2020 and September 2020 to November 2020 participated in the curriculum. Residents in their fifth year of training were invited—but not required—to attend. Residents who consented to the use of their data for research purposes were included in the study.

Curriculum

The structured, longitudinal POCUS curriculum comprised assigned online POCUS lectures,20 live model scanning sessions, video reviews, summative assessments, and a portfolio of supervised non-clinical and clinical scans (Appendix 1). The curriculum was developed after a needs assessment and extensive review of the anesthesia POCUS and CMBE literature with input from local echocardiology and emergency medicine colleagues. The curriculum development and implementation process are detailed in Appendix 2.

The POCUS applications taught included rescue echo, lung ultrasound, focused assessment with sonography in trauma (FAST), abdominal aorta, airway, and gastric ultrasound. The curriculum prioritized rescue echo with dedicated practice and assessment at every session. Rescue echo20 is a systematic approach to the assessment and management of perioperative hypotension and/or hypoxia using five standard two-dimensional focused cardiac ultrasound views: parasternal long axis view, parasternal short axis view, apical four-chamber view (apical 4), subcostal view, and inferior vena cava view,22 as well as bilateral lung ultrasound for pneumothorax. Comprehensive lung ultrasound, FAST, abdominal aorta, airway, and gastric ultrasound were formally covered and assessed during two live scanning sessions per year (Appendix 1).

Residents were assigned 20 hr of flipped classroom video lectures20 and journal article reading, and attended eight sessions (20 hr) per academic year. Nonclinical scans on simulated patients (SPs) without pathology were conducted in the clinical learning and resource center procedure labs weekly for two hours, followed by 30 min of video review. A mean number of four trainees (range, 2–8) attended each session with two bedside instructors, four ultrasound machines (GE Venue 40 & 50, GE, NY, USA), and four SPs. An anesthesiologist and a professional cardiac sonographer provided bedside instruction and assessment at each session. The registered diagnostic cardiac sonographer (RDCS) providing bedside instruction and image acquisition scores had 19 years of clinical experience. There were eight POCUS-trained anesthesiologists who provided the bedside instruction and assigned entrustment scores; of these, two had completed a critical care fellowship that included POCUS training, two had an echocardiography fellowship, one had a cardiac anesthesia fellowship, and one had a trauma fellowship. Five of the faculty were cardiac anesthesia providers, credentialed by the National Board of Echocardiography in Advanced Perioperative TEE. Additionally, seven faculty had undertaken the echo-guided life support23,24 training, three of the faculty evaluators had Canadian Point of Care Ultrasound Society25 core and/or acute care resuscitation independent practitioner certification, and three had a perioperative and critical care echo certificate. Image acquisition assessment tools are provided in the Electronic Supplementary Material (ESM; eAppendix).

Video review of pathology, group reading, and case discussion at the end of each scanning session drove learning and allowed trainees’ knowledge, image interpretation, and integration into clinical management to be assessed so the summative curricular entrustment scores reflected more than image acquisition. The video review component of the curriculum was flexible, with residents and faculty bedside instructors encouraged to bring cases or use a free website such as The POCUS Atlas (https://www.thepocusatlas.com/).

During video review, residents in the small group sessions interpreted and discussed videos from cases with pathology. Image interpretation with a binary approach where possible, of “present/absent (or unable to determine)”, was used to correctly identify:

(a) Rescue echo—hemodynamic states: normal, hypovolemia, low afterload, left ventricular failure, right ventricular failure, pericardial effusion or tamponade physiology, gross valvular abnormalities, basic features of dynamic left ventricular outflow obstruction, signs of chronic cardiac disease, and pneumothorax20,26,27,28

(b) Lung ultrasound—pneumothorax, interstitial syndrome, and pleural effusion

(c) FAST—presence of free fluid (abdominal or pericardial fluid)

(d) Abdominal aorta (AA)—presence of an abdominal aortic aneurysm

(e) Airway ultrasound—anatomy and cricothyroid membrane (CTM)

(f) Gastric—gastric contents: empty, clear fluid, or solids.

Portfolio entries for curricular scans included date, POCUS application, sonographer, faculty instructor, and the image acquisition and entrustment score collected for each participant from the curricular assessments. Portfolio entries for scans in the clinical setting included date, type of POCUS exam, location, clinical scenario, purpose of the exam, faculty supervisor, and their assigned entrustment score. Residents were encouraged, but not required, to log their POCUS scans similarly in a Health Information Protection Act-compliant platform, such as a personally curated logbook or using readily available POCUS logging online apps.

Outcomes

The primary outcome was the number of supervised scans and sessions required to achieve average scores ≥ 4 (“may use independently”). Faculty evaluators completed a POCUS skills entrustment scale15 at each scanning session. Also included were clinical POCUS entrustment scores from participants’ Royal College of Physicians and Surgeon’s Core EPA #613 (ePortfolio) completed by the supervising faculty.

The POCUS-specific domains evaluated on the five-point entrustment scale for each application were knowledge and insight, image acquisition, image interpretation, and integration into patient management.15 The assigned overall entrustment score reflected the lowest score obtained in any domain for the observed POCUS application. Entrustment scores ranged from 1 to 5, where 1 = “Staff had to do/resident may not use ultrasound,” 2 = “Staff had to talk resident through/resident may use ultrasound only under direct supervision,” 3 = “Staff had to prompt resident from time to time/resident may begin to use ultrasound with indirect supervision,” 4 = “Staff needed to be in room just in case/resident may use ultrasound independently,” and 5 = “Staff did not need to be there/resident is independent.”

Secondary outcomes included image acquisition skills at each session and retention of rescue echo competency. We planned to measure retention after three months, but the COVID-19 pandemic resulted in a ≥ six-month pause in curricular sessions.

Image acquisition was assessed by an experienced registered diagnostic cardiac sonographer (or the faculty evaluator for non-echo applications) using image acquisition and assessment tools modified from Gaudet et al.29 (eAppendix, ESM) at each session. Image acquisition scores were timed, with a five-minute maximum.

Assessment of insight, image interpretation, and integration into management in curricular sessions relied on observations from POCUS session video review of pathology and case discussion and was reflected in the overall entrustment score.

Residents’ ongoing portfolio development entrustment scores were deidentified and recorded in an Excel spreadsheet for the curricular and clinical scanning. An electronic questionnaire was distributed to participants to capture demographic characteristics, previous POCUS experience, and evaluation of curriculum. Data were collected from January 2019 to 6 November 2020, except during the summer holiday and pandemic pause described.

Analyses

Entrustment scores were averaged among learners for each session and used to identify the learning curve (i.e., number of supervised scans vs entrustment scores). Scores were also fit to the regression curve to determine the number of supervised scans and sessions required to achieve mean entrustment scores ≥ 4. We used Cronbach’s alpha to determine the internal consistency of the items in each section or subsection of the image acquisition and assessment tools.29 We used Eta-squared (η2) to calculate effect size and Welch’s t test to calculate P values for entrustment and image acquisition score improvements.

Results

Thirty anesthesiology residents participated in the study, and data are reported from 29 participants; one participant was withdrawn because they completed their residency before data could be collected. Participants, according to their year entering the curriculum, comprised 14 second-year residents, ten third-year residents, and five fourth-year residents reporting a range of POCUS experiences prior to the curriculum (Table 1).

Table 1 Participating residents’ mean (range) number of previous POCUS scans
Full size table

Entrustment scores

Rescue Echo

There were 194 entrustment scores collected from the curricular sessions and 22 entrustment scores from RC “cardiopulmonary” POCUS (EPA 3.6) that were averaged for trainees according to the number of scans. The mean score at their first curricular session was 2.7 (“may use only with direct supervision”). The mean (standard deviation [SD]) number of directly supervised scans and sessions required for trainees to reach a mean entrustment score of ≥ 4 was 36 (10) supervised scans over nine sessions (13 months) for rescue echo (η2, 0.46; 95% confidence interval [CI], 0.21 to 0.63; P = 0.001) (Fig. 1). The mean entrustment scores assigned in curricular and clinical settings are separated into two learning curves in Fig. 2.

Fig. 1
figure 1

Rescue echo entrustment score learning curve

Full size image
Fig. 2
figure 2

Rescue echo entrustment score learning curves in curricular and clinical settings

Full size image

Lung ultrasound

Thirty-three curricular entrustment scores were collected with a mean at the first curricular session of 3.1 (“May use with indirect supervision”). The mean (SD) number of supervised scans and sessions required for trainees to reach mean entrustment scores of ≥ 4 was 8 (3) scans over two sessions (η2, 0.01; 95% CI, 0.001 to 0.12; P = 0.43). Lung ultrasound entrustment scores from the clinical setting have not yet been collected.

Entrustment for independent use was not achieved for FAST, AA, airway, or gastric ultrasound. For FAST and AA, the average entrustment score of study participants was 3.0 after eight supervised scans over two sessions, which was not notably different from baseline. The average entrustment score of study participants was 2.0 for gastric and airway POCUS after eight supervised scans over two sessions.

Image acquisition scores

We collected 281 timed image quality scores: 172 for rescue echo, 33 for lung ultrasound, 20 for airway ultrasound, 39 for FAST, 17 for AA, and 0 for gastric POCUS. Mean image acquisition scores for rescue echo improved from 35/60 (58%) at baseline to 52/60 (87%) after 36 scans (η2=t, 0.31; 95% CI, 0.08 to 0.50; P = 0.006) (Fig. 3). Mean image acquisition scores for lung ultrasound improved from 32/36 (90%) at baseline to 34/36 (94%) after eight supervised scans (η2, 0.02; 95% CI, 0.001 to 0.13; P = 0.35). There were insufficient data to report image acquisition results for FAST, AA, gastric, and airway applications, for which competence was not yet achieved.

Fig. 3
figure 3

Rescue echo image acquisition scores

Full size image

Retention

Following a six-month hiatus from supervised curricular scanning due to the COVID-19 pandemic, we obtained retention data from 11 participants. No statistically significant conclusions can be made for POCUS retention because of the modest sample size. The rescue echo entrustment score learning curves obtained before and after the six-month hiatus due to the pandemic are displayed in Fig. 4.

Fig. 4
figure 4

Rescue echo entrustment score learning curves before and after COVID-19 pandemic delayed supervised POCUS scanning for six months. POCUS = point-of-care ultrasound

Full size image

Curriculum evaluation

Most residents agreed or strongly agreed that the POCUS applications taught in the curriculum were relevant to the practice of anesthesiology (20/23; 87%), that the University of Utah online anesthesiology assigned POCUS lectures20 were relevant and appropriate as "flipped classroom" content for the resident POCUS curriculum (20/23; 87%), and that having a professional echo-sonographer as one of the bedside instructors was a valuable component of the POCUS curriculum's live-scanning sessions (20/23; 87%).

Discussion

This study identified the learning curve for rescue echo and lung ultrasound using CBME entrustment scores assigned by POCUS-trained anesthesia faculty with standardized patients in a simulated clinical environment and on real patients in the clinical setting. Anesthesiology residents, with a range of previous POCUS experience, required 36 supervised rescue echo scans (14.5 hr supervised dedicated rescue echo curricular scanning) over 13 months (9 sessions) to achieve competence. This finding is in line with a variety of professional groups’ recommendations for training that includes a theoretical component as well as an apprenticeship ranging from 30 to 50 cardiac POCUS scans.25,28,30,31,32 Competence was not yet achieved in FAST, AA, airway, and gastric ultrasound by participants in our study. These applications were afforded only two dedicated curricular scanning sessions per year, and likely require more frequent, longitudinal supervised scanning to develop competence.33 Our anesthesia resident POCUS curriculum and results are in line with those described in the recent consensus-based Canadian recommendations for training and performance in basic perioperative point-of-care ultrasound32 with two exceptions: (1) lung ultrasound mean entrustment scores were ≥ 4 after only eight supervised scans in our study in contrast to 15 scans suggested in the recommendations, and (2) competence was not achieved in airway ultrasound after eight supervised scans, although the recommendations suggest that competence can be achieved after five supervised scans. Trainees reported an average of five lung ultrasound scans prior to participation in the curriculum (Table), and lung ultrasound was practiced as part of the rescue echo sequence in our study. Participants’ previous experience and concurrent uncaptured lung scans may have contributed to the lower number of scans required to achieve competence in our study. Although the participants in our study achieved a mean entrustment score ≥ 4 after a mean (SD) of 8 (3) supervised lung scans, the effect size and P value were not significant. This is likely because of the higher baseline scores upon entering the curriculum and the lower number of observations for lung ultrasound in our study.

Our study results should not be applied to any one learner but can provide guidance for allocation of resources for optimal programmatic efficiency in POCUS training. Clarifying the progression of learning and the average number of sessions and scans required to achieve competency may help instructors and residency training programs develop and design their own POCUS training curriculum. Our retention data suggests that POCUS skills may be retained after a six-month hiatus, but more participant data are required to confirm this.

A strength of this study was the consistent, objective scoring of the rescue echo image acquisition skills by an RDCS and scoring of entrustment by a small group of POCUS-trained faculty. This is in accordance with the Focus cardiac ultrasound core curriculum and core syllabus of the European Association of Cardiovascular Imaging,28 which stated that focused cardiac ultrasound recommendations should include reference echocardiographic community representatives in concert with respective specialities. Trainees identified the sonographer as a significant strength of the POCUS curriculum. Adding a general sonographer for the noncardiac POCUS applications could also be valuable to the POCUS curriculum and has been recently incorporated into our curriculum. This also reduced costs by reducing the need for additional faculty compensation and freeing other faculty for clinical duties.

The study was conducted under real curricular and clinical training conditions, which included the presence of faculty instructors with an array of POCUS backgrounds, residents with a wide variety of POCUS experience, conflicting clinical duties or out-of-province electives occasionally interrupting the frequency of sessions for participants, and obstacles to clinical integration.

Strengths of our curriculum include our College of Medicine’s Simulated Patient Program34 (including standardized patients, space, machines, administrative and support staff), which is a valuable resource that supported our curriculum. The POCUS curricular sessions were scheduled at the same time as the long-established resident weekly academic half-day to minimize clinical interference but with the trade-off of residents missing approximately one in four academic half-day lectures to attend the POCUS small group sessions. The collaboration with our local emergency medicine and cardiology colleagues allowed the curriculum to benefit from integrating the well-established and rigorous emergency medicine POCUS training31 and local echocardiology expertise with anesthesia-specific training needs. The curriculum considered the ultrasound training already established in our anesthesia residency training program. It also identified perioperative POCUS training needs and reflected the perioperative differential diagnosis of hemodynamic instability20,26,35,36,37 and scanning techniques for anesthesiologists in the perioperative environment. The experienced sonographer ensured residents learned proper scanning techniques while the experienced faculty introduced a layer of anesthesia-specific POCUS pragmatism to the curricular scanning sessions. As residents gained confidence, they were encouraged to simulate real-life challenges of intraoperative scanning including varied ultrasound machine positioning, patient positioning, trainee positioning, and obtainable views. The timed image acquisition assessment at each session reinforced the importance of a methodical, focused approach and simulated the pressure of using POCUS to answer a clinical question about an unstable patient.

There are several limitations of our study, most notably the low number of clinically observed POCUS encounters relative to the number of curricular nonclinical observations and lack of a validated summative assessment. Ideally, competency should be assessed based on multiple observations and in a variety of clinical scenarios.10 There were many obstacles to clinical POCUS and clinical assessments, including a lack of ultrasound machines, probes, and an archiving system, insufficient number of POCUS-trained faculty, and time-pressures of the operating room. Another significant barrier to clinical data in our study was the resident POCUS curriculum overlapping with the national RCPS anesthesiology training program’s transition to Competence by Design (CBD), with the first cohort of POCUS trainees (13/29 participants) not being CBD residents and therefore not being assigned clinical entrustment scores. Additionally, during the transition and early CBD roll-out, our Saskatchewan anesthesiology residency program and competency committee reported lower rates of resident evaluation return for all EPAs, which was attributed to changing online evaluation platforms and lack of familiarity with CBD. Resident enthusiasm to integrate their POCUS skills into clinical practice prompted the implementation of an anesthesia resident clinical POCUS elective in August 2020 which, in addition to a more mature CBD program, resulted in increased clinical POCUS assessments. Ten of the 16 CBD participants currently enrolled in the curriculum have been deemed competent by the competence committee, having achieved the POCUS EPA #3.6 based on clinical assessments, which lends validity to the entrustment scores assigned in the simulated setting.

This was an observational study, and the methodology may be considered a limitation. There is, however, accumulating evidence for improved learning and patient outcomes with simulation enhanced medical training.6,38,39 The use of simulation-based medical education (SBME) and deliberate practice has shown improvements in skill performance and better learning outcomes than clinical teaching for many similar medical skills such as bronchoscopy, laparoscopy, management of shoulder dystocia, and central venous line insertion.38

Although this is a resource-intensive curriculum that could not have been developed or delivered without the local simulation resources, academic funding, tremendous support from local POCUS champions, and contributions from our echocardiology and emergency medicine colleagues, we suspect many academic centres with residency training programs would have access to similar resources and local experts.

Conclusion

This study is the first to use entrustment score data to show that competency in rescue echo (focused cardiac ultrasound and lung ultrasound for pneumothorax) and lung ultrasound can be developed during residency through a structured longitudinal curriculum and suggests an approximate time and number of scans required to achieve competency. These results can be used as a foundation for an evidence-based, permanent anesthesiology POCUS curriculum for residents.

Future research will determine if competence is retained and the number of scans required to achieve and retain competence in FAST, AA, gastric, and airway POCUS applications. There remains a need for development and validation of rigorous and comprehensive speciality-specific standardized summative assessment tools. Future investigations should also explore how residents and faculty incorporate POCUS into their clinical practice and whether the use of POCUS by anesthesiology residents and faculty affect patient outcomes.