Evaluation of a new goal-directed training curriculum for point-of-care ultrasound in the emergency department: impact on physician self-confidence and ultrasound skills

Purpose Developing countries need effective and efficient training curriculum for the point-of-care ultrasound (POCUS) in the emergency department (ED). We have developed a new goal-directed training curriculum focusing on critical POCUS procedures used in ED. Methods To evaluate the impact of the new POCUS curriculum on ED physicians’ performance/self-confidence, we carried out a quasi-experimental trial at ED training center of Peking Union Medical College Hospital, in which we trained two groups of physicians using either traditional curriculum or the new goal-directed curriculum. We measured the confidence in performing Focused Assessment with Sonography in Trauma, thoracic, vascular, ultrasound-guided puncture, echocardiography and undifferentiated shock diagnostic ultrasound at baseline, training completion and 1 month after training. We also measured the performance skills at the time of training completion. The relative value of the new curriculum was evaluated by differences in the confidence and performance measurements, with control for baseline measurements and confounding characteristics in univariate analyses and multivariate linear regression models. Results After training, both groups of trainees reported significantly increased self-confidence in performing POCUS as compared with baseline. Trainees with the new curriculum had statistically significant higher self-confidence increase and performance scores (p < 0.05), e.g., increase in confidence for diagnosis of undifferentiated shock at training completion = 3.14 vs. 1.85, 1 month after training = 2.22 vs. 1.56, for new and traditional curriculum, respectively. The new curriculum also resulted in a higher number of shock POCUS done within 1 month after training: 1.73 vs. 0, and higher overall performance scores: 165 vs. 113. The findings were robust when controlling for imbalanced baseline characteristics in multivariate regression models. Conclusion We conclude that a goal-directed, intensive but brief ED POCUS curriculum significantly increases trainee self-confidence, performance, and promote trainees to perform more procedures.


3
Background Many clinical settings have adopted the point-of-care ultrasound (POCUS) in practices [1][2][3][4][5]. Recognizing the value of the POCUS and the need for adequate training, various clinical settings have developed POCUS training curriculums [6][7][8][9][10][11]. The unique characteristics of emergency clinical workflow require dedicated training for POCUS in the emergency department (ED). ED physicians need to perform the image acquisition on the spot and extract information in a timely fashion to aid the patient diagnosis and treatment. On the other hand, the application of POCUS in ED can be intensely focused on the specific use cases and often driven by common goals of emergency medicine (EM) patient management.
Although various EM POCUS training guidelines have been developed [12][13][14], POCUS training curriculums in ED remained quite varying between institutions, as recognized by the International Federation for Emergency Medicine in 2015 [6,15]. Typical curriculums usually take several days including hours of didactic training followed by practices on proctored scans. The lengthy sessions of studying the course materials, interactive online learning, modular learning, and hands-on experiences are to ensure the trainee will acquire the knowledge and skills to efficiently utilizing POCUS. However, time-consuming sessions are expensive to institute, especially for developing countries with a considerable gap in physician POCUS expertise and limited resources. Curriculums that delivers both efficiency and quality are in great need to enable trainees in achieving a solid understanding of the anatomic structure, obtaining high-quality images in standard views, recognizing pathological findings and supporting clinical decisions.
With the aid of new technology of 3-D ultrasound image recording, we have developed a new, brief, goal-directed curriculum for POCUS training in ED. We expect the goaldirected curriculum, requiring the same amount of training time, will outperform the previous standard curriculum by improving trainee self-confidence and increasing performance carrying out POCUS. This study aimed first to introduce the new brief POCUS curriculum and then to evaluate the impact of the new curriculum on trainee performance/ self-confidence in relative to the standard curriculum.

Study design and participants
This study was a quasi-experimental trial carried out to investigate the impact of two different POCUS curriculums on ED physicians' performance/self-confidence. We conducted the trial at the ED training center of Peking Union Medical College Hospital (PUMCH). We recruited participants enrolled in PUMCH emergency POCUS training course from 28 December 2015 to 17 June 2017. Trainees enrolled before 31 December 2016 underwent training using the traditional curriculum, and those enrolled after received training using the new curriculum. As the enrollment to the PUMCH bedside ultrasound training course was open to all interested emergency physicians nationwide, there was no requirement for prior ultrasound education or hands-on experiences, and no specific control for trainees' allocation to either the traditional or the new goal-directed curriculum.
We approached the two consecutive cohorts of trainees for consent and included only consent trainees in the study. We further excluded trainees who failed to finish the course or/and did not respond to follow-up. The institutional board of PUMCH reviewed and approved the study, considering this study presented no more than minimal risk to subjects.

Study interventions
For both standard and new curriculums, trainees participated in a two and half day training course in which they formed study groups of about four to seven members. One or two teaching physicians were responsible to coach individual study group in hands-on sessions.

Traditional curriculum
Subjects enrolled in the training program before 31 December 2016 received training using a traditional curriculum. The curriculum started with an introductory lecture on the fundamental ultrasound knowledge, which covered physics and instrumentation of emergency ultrasound, followed by lectures on how to obtain standard ultrasound views without emphasis on relevant ultrasound anatomy, or normal/ pathological images interpretation. After the educational lectures, the program provided hands-on practices on simulated manikins for trainees to obtain standard views with the guidance of instructors. The curriculum completed with the introduction of Rapid Ultrasound for Shock and Hypotension (RUSH) protocol and a lecture for using POCUS to aid the diagnosis of undifferentiated shock.

New goal-directed curriculum
Subjects enrolled in 2017 underwent training using an improved, goal-directed curriculum. We identified the five core applications that ED physicians commonly perform to assess patients in their anatomical structures and underlying pathophysiological changes: Focused Assessment with Sonography in Trauma (FAST), thoracic, vascular, ultrasound-guided puncture (UGP) and echocardiography. Details of the five ultrasound applications are included in Appendix 1. The learning objectives of the training were to facilitate trainees to correctly carry out the above five ultrasound applications and obtain accurate assessment through a limited number of standard views. Also, we retained our emphasis on the POCUS procedure for undifferentiated shock diagnosis and introduced the "Tamponade/Tension, pneumothorax, Heart, Inferior vena cava, Respiratory system, Deep venous thrombosis/Aorta Dissection" (THIRD) procedure in the new curriculum [16].
Before formal courses, we provided trainees with precourse learning materials to familiar with the necessary knowledge of ultrasound and the consensus of evaluation for undifferentiated shock patients. The pre-course materials contained reading literature and web-based video sessions. The ED of PUMCH has published details about the pre-course learning in May 2017 [16].
During the formal course, we improved the first introductory lecture for basic ultrasound knowledge by integrating hands-on practice with coaching, which included ultrasound probe holding, probe placement/movements, understanding of three ultrasound plains of tridimensional organ, explanation on four sides of the view, and summary on six steps of obtaining a high-quality view.
After the introductory lecture on basic ultrasound knowledge, we also changed the sequence of subsequent training sections with the improvement of contents and technologies. Subjects had the first lecture on the introduction of anatomy with 3D video records and photos of dissecting real organ. The second lecture provided the tips to gain a standard view in details and compared 3D images with normal ultrasound views. Trainees practiced drawing ultrasound views on simulated manikins after this session. Each trainee was assigned to a group for hands-on practice instructed by group instructor, the ratio of instructor to trainee was 1:7. Once the trainee performed required standard view on the manikins, simulated 3-D images were presented to the trainee to demonstrate various pathological findings under that particular standard view. We also provided standard patients with no pathological conditions to trainees alongside with the simulated manikins. Subjects had the last lecture on the interpretation of pathological ultrasound images with the real clinical case.
The final step of the new curriculum was to review and practice in groups all critical ultrasound diagnostic/ treatment procedures (FAST, thoracic, vascular, UGP, and echocardiography) with emphasis on the THIRD protocol for undifferentiated shocks. Instructors presented simulated case scenarios and corresponded ultrasound findings while trainees were obtaining images on models.

Measurements and outcomes
We conducted a baseline survey of study participants before training for basic demographics and prior ultrasound experiences, which include seven single choice questions and one open-ended question. The participants were requested to rate for the self-confidence in performing of five core applications: FAST, thoracic, vascular, UGP, echocardiography and diagnosing undifferentiated shock (RUSH for old curriculum or THIRD for new curriculum). The confidence measurements ranged from 1 to 5 with 1 being the least confident and 5 being the most confident.
Multiple course instructors assessed trainees' performance in carrying out the ED ultrasound procedures (FAST, thoracic, echocardiography, and UGP) after training. To give a fair assessment for trainees enrolled in the study, we assigned course instructors to evaluate the performance of trainees who they were not in direct coaching with during hands-on and group sessions. However, we could not completely blind the instructors of the trainees' seniority and experiences. Instead, we specifically required an objective assessment of the trainees' skill level using an anchored rating scale as the assessment tool: We assign score of one to participants who were unable to obtain a view and unable to explain normal image even with extensive instructor's coaching; three when they were able to obtain a low-quality view and give some of the image interpretation with some coaching; five when they were able to gain a standard view with high image quality and make accurate image interpretation without assistance. The total performance score summing up the 4 subcomponents, 36 items ranges from 36 to 180. All course instructors received prior standardized training in the right using of the assessment tool.
One month after the completion of the training, we contacted trainees about the reassessment of self-confidence and post-training ultrasound practices. We chose to follow-up on trainees' self-confidence 1 month after the training because we believe by that time trainees would have had enough time to digest the learning in the course and make enough attempts to practice POCUS independently.
This survey was administered online using the research electronic data capture (REDCap) medical data system.

Statistical analysis
An independent statistician analyzed the data. The primary outcome was trainees' performance measured after the training. Secondary outcomes included self-confidence measured at the completion of training and 1 month after. To control for the differential distribution of baseline self-confidence between groups, we calculated the differences between the post-training and the baseline self-confidence measurements. To compare the traditional curriculum and new goal-directed curriculum, we tabulated trainees' baseline characteristics and outcomes by training groups, with means and standard deviations presented for continuous variables, and counts and percentages for categorical variables. We carried out univariate statistical comparisons between training groups using the two-sample t test supplemented by the two-sample Wilcoxon rank sum test (given the small sample size and skewed distributions of variables) for continuous variables, and the Fisher's exact test for categorical variables.
We also performed multivariate linear regression analyses to examine the independent association between training programs and outcomes while controlling for the potential baseline confounders. Only imbalanced baseline characteristics between the two trainee cohorts were included in the multivariate regression model (either p value < 0.05 or relative differences greater than 10%). The statistical significance threshold was set at p < 0.05. Individuals with missing data were excluded from analysis. All data analyses were performed using STATA 12.0 (StataCorp. 2011. Stata Statistical Software: Release 12. College Station, TX: StataCorp LP.).

Patient and public involvement
Our study participants were not considered as patients, and there was no other public involvement other than the trainees enrolled in PUMCH emergency POCUS training course. The study's results were disseminated to the participants who chose to receive updates after.

Overall sample
During the two consecutive training programs, we enrolled and consented 66 trainees from over 20 hospitals throughout China. We excluded two trainees who did not finish the course for personal reasons. Twenty-seven trainees enrolled before 2017 received traditional POCUS training, while 37 trainees received training using the new curriculum.
On average, trainees had 5.58 years of practice experiences in ED, and 51.6% had some prior ED ultrasound training. Many indicated that they had performed ED ultrasound before, with the most frequently performed being ultrasound-guided puncture (45.3%), followed by vascular (37.5%), echocardiography (28.1%), and the least performed being diagnostic procedures for undifferentiated shock (7.8%). Similarly, the prior experiences with different ultrasound procedures were also reflected in the corresponding baseline self-confidence, with diagnosing undifferentiated shock being the one with the lowest confidence. However, a small proportion of them indicated that they were independently performing ED ultrasound before training (9.4%) ( Table 1).
After training, all trainees reported significantly increased self-confidence in performing ED ultrasound as compared with baseline. One month after training completion, the confidence tended to decrease moderately from the end of training but was still higher than the baseline measurements. For example, confidences in performing shock protocol were 1.50, 3.44 and 2.59 for baseline, training completion and 1 month after training, respectively (Tables 1, 2).

Trainees' baseline characteristics by training curriculums
Trainees received traditional/new training programs that differ in several baseline characteristics. Trainees with new curriculum tended to have more years practicing in ED (7.41 vs. 3.07, p < 0.001), more likely to have prior ED ultrasound training (64.9% vs. 33.3%, p < 0.001), performed slightly more ED ultrasound before (total amount: 17.35 vs. 13.70, p = 0.208; counts of types of ED ultrasound: 2.03 vs. 1.19, p = 0.060). Trainees who received the new curriculum also had higher baseline self-confidence measurements than those who received traditional curriculum (Table 1). However, trainees with new curriculum are less likely to indicate they were independently performing ED ultrasound (8.1% vs. 11.1%) with an insignificant p value of 0.691 at the threshold of 0.05.

Trainees' post-training measurements by training curriculums
Trainees with new curriculum had statistically significantly higher measurements in self-confidence after training, reflected in numbers both at training completion and at 1 month after training. Even after adjusting for the differential distribution of baseline self-confidence by calculating the absolute delta of post-training confidence from baseline, the new curriculum was associated with higher trainee selfconfidence (post-training vs. baseline and 1-month posttraining vs. baseline). For example, in FAST performance, the confidence measurements were 3.92 (new) vs. 3.37 (traditional) for post-training, 3.27 vs. 2.26 for 1-month posttraining, 2.03 vs. 1.78 for increased confidence post-training as relative to baseline, and 1.38 vs. 0.67 for increased confidence 1-month post-training as relative to baseline.
Trainees with new curriculum also ended up performing more ultrasound post-training than trainees received traditional curriculum (14.2 vs. 1.44, p < 0.001) and had higher performance scores at training completion (165.76 vs. 113.44, p < 0.001).
When adjusting for potential confounding factors such as imbalanced baseline characteristics and baseline confidence levels, the new curriculum was consistently associated with higher increases in self-confidence measured at either training completion or 1 month after (except for ultrasoundguided punctures), more substantial number of ultrasound performed in 1-month post-training, and higher performance scores at training completion (Table 3).

Discussion
Bedside ultrasound has become a vital tool to aid diagnosis and treatment in the ED. Better utilization of bedside ultrasound has also been associated with improved patient outcomes and decreased costs in the ED [2,[16][17][18][19][20]. However, multiple studies have recognized a gap of POCUS utilization in ED and identified the lack of adequate training as the primary barrier for effective usage of POCUS [21,22]. In a 2014 national survey of US critical care fellowship directors, the authors indicated even in a country like the United States, which have developed guidelines for POCUS decades ago, there were significant variations of the existing curriculums and a dire need for an improved curriculum to close the gap in POCUS training [4]. In developing countries in which resources are scarce, ED physicians face more challenges in adopting POCUS in their daily practices.
First, our study quantitatively assessed the gap of POCUS expertise in Chinese ED physicians. Although many trainees had prior experiences performing POCUS before the training, a tiny proportion of them indicated that they were independently performing ultrasound examinations, and on average, the self-confidence levels were all lower or close to a rating of two ("no confidence but willing to try") before training. When it comes to complex bedside ultrasound procedures such as RUSH or THIRD, It is not surprising that only 7.8% of trainees had ever performed POCUS to aid diagnosis of undifferentiated shock in ED practice. At the time of the study, most Chinese ED programs relied primarily on informal bedside teaching for ED POCUS without a proper ultrasound curriculum in place. The training team is rarely staffed with enough experienced course instructors and uses outdated teaching materials.
Second, this study has demonstrated that a two and a half day ultrasound course using a new curriculum can effectively train ED clinicians in POCUS skill. Compared with the traditional training curriculum, trainees with the new curriculum had higher performance ratings even after adjustment for multiple confounders. The new curriculum was also associated with increased self-confidence at the completion of the training, which persisted when re-evaluated in 1 month after. Interestingly, the treatment effect (the new  curriculum vs. the traditional) in the delta of self-confidence was of greater magnitude when measured in 1 month after training, compared with that measured at training completion. We hypothesize that the new curriculum allowed trainees with greater confidence after training completion and encouraged them to perform more POCUS procedures, which snowballed and provided additional positive feedbacks in increasing ones' self-confidence. Third, we designed and tailored the new curriculum to meet the greater need for effective and efficient ED POCUS training programs in low-and middle-income countries like China. Compared with North American and European countries, developing countries face more challenges of limited resources and heavy patient workload. Curriculums in developed countries usually mandate a low student to instructor ratio (no higher than 5-1), may require pathology models to include standard patients with pathological conditions, and usually take a longer time to complete (half-day for a single application) [6,15,23]. When developing the new curriculum, we have integrated other countries' experiences of designing a POCUS training program in a limited resource setting [8,21,[24][25][26]. We have identified the commonly utilized POCUS procedures in Chinese ED and designed the intense training sessions accordingly. Our curriculum was shorter, more focused on the selective list of procedures, and maintained relatively high students to instructors' ratios.
We believe that the increase in performance rating, number of POCUS performed after training, and improved selfconfidence can be attributable to multiple improvements we made to the curriculum. The main change of the new curriculum was the inclusion of the procedure-focused training for Fast, Thoracic, vascular, USP, echocardiography and undifferentiated shock. Trainees were trained to operate ultrasound on standard manikins and patients, perform the above standard procedures while presented with simulated clinical scenarios. We believe the simulated scenarios are more reflective of a real clinical setting and boost the retention of new skills and information with lower technique requirement on simulating resource [27].
In the new curriculum, we experimented with the concept of the flipped classroom by providing the teaching materials and instructional contents before the didactics. The utilization of flipped classroom provided the opportunity for more time spent in the didactics reviewing and coached practices so that the curriculum could achieve better results within limited training time [23]. Also, we believe inserting handson practice before the lecture of pathological images also helped trainees better understand the anatomical structure, so that they could relate the ultrasound maneuvers to abnormal findings on the images obtained. As noted in previous guideline [13,23,28], rotating the skill learning and didactics sessions also improves the efficiency of the training program. Lastly, the availabilities of interactive, 3-D images made it possible to present simulated pathological findings while trainees were practicing standard views on manikins, without having real standardized patients available for every pathological condition.
There are several limitations to our study. There was no randomization of trainees' allocation to the old or new curriculum, which resulted in the differential distribution of baseline characteristics of trainees. Especially for trainees' performance measurement, we were not able to collect the baseline measurement to be controlled for. However, the treatment effect of our intervention on outcomes remained robust when evaluated as a delta from baseline, or in multivariate regression models adjusted for the imbalanced baseline characteristics. We acknowledge the potential existence of residual confounding. Its relative small sample size also limits this study, and the training curriculum was only developed and evaluated in a single institution. However, our recruitment of trainees was open to hospitals nationwide, and our results confirmed a broad distribution of physicians with different seniority and experiences of POCUS. We expect our findings to generalize well in other institutions. In our study, we have not directly measured the curriculums on all the Kirkpatrick's four level evaluation matrices. We have focused more on the behaviors and results aspects by measuring the changes in self-confidence performing POCUS, and number of cases performed after the training. Future researches are going to provide more detailed information revealing the subjective reactions of the trainees, and the reasons why they have/have not achieved the learning goal. Those additional measurements will provide further insight to find the root cause of success and failures in teaching POCUS at the bedside in ED, and help improve the new curriculum.

Conclusion
This study suggests that a goal-directed, intensive but brief (2.5 day) ED POCUS curriculum is associated with a significant increase in trainee self-confidence and performance. The focused training on critical POCUS procedures commonly used in the ED setting has contributed to improved ED POCUS skills. Further study is needed to identify the optimal allocation of time between lectures and hands-on practices and explore other enhancements for the curriculum.
Author contributions Di Shi was the major contributor to write the manuscript; Jihai Liu made a contribution to the study design and collected data; Jun Xu analyzed the data; Huadong Zhu and Xuezhong Yu interpreted the data. All the authors read and approved the final manuscript.
Funding This work was supported by CAMS Fundamental research fund for Central public welfare research institute, Grant number 2018PT32004.

Compliance with ethical standards
Conflict of interest The authors declare that they have no competing interests.

Ethics approval
The institutional board of Peking Union Medical College Hospital (PUMCH) reviewed and approved the study, the address of the approving ethics committee is No. 1 Shuai Fu Yuan, Dong Cheng District, Beijing, China, 100,730.

Availability of data and material
The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creat iveco mmons .org/licen ses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.