Critical care ultrasound (CCUS) previously focused on vascular access, but has an ever-increasing role, including imaging of the heart, lungs, and abdomen. Critical care ultrasound offers potential assistance in patient diagnosis and treatment that is point-of-care, real-time, and goal-directed. Putative advantages of CCUS include enhanced diagnostic accuracy in hemodynamic instability,1 reduced procedural complications, increased cannulation success,2,3 and more accurate diagnosis and therapy following respiratory failure.4 Therefore, if widely accepted, CCUS could substantially improve critical care delivery, efficacy, and safety. Accordingly, CCUS been endorsed for critical care medicine (CCM) practitioners by 13 critical care societies, including the Canadian Critical Care Society.5,6

Critical care ultrasound requires knowledge, hands-on training, and skills maintenance. As a consequence, many frontline CCM practitioners (both trainees and attending physicians) have yet to achieve basic competency or integrate it into their practice.5,7,8,9 While published recommendations 5,6,10 provide expert guidance on pathways for CCUS training, we should ensure that those recommendations work for clinicians. Published surveys from U.S. CCM training programs have highlighted that many institutions have no deliberate strategy, no formalized curriculum, no trained faculty, and insufficient engagement of frontline practitioners.7,11 A coherent educational strategy is needed to build widespread fluency across CCM practitioners.

The needs assessment is widely accepted as a foundational step in designing any educational intervention.12,13 It provides knowledge and shapes new curricula by identifying gaps, characterizing barriers, prioritizing needs, and assessing resources.12,13 Further, a needs assessment can be transformative because it engages all potential learners as active collaborators in the creation of their own education.14 In short, a CCUS curriculum should be informed by the literature, but should also strive for stakeholder engagement and broad acceptance. Accordingly, we performed a mixed-methods needs assessment of frontline CCM practitioners. Our goal was to establish a curricular framework that could be universally applied across hospitals.

Methods

Study design and setting

We performed a mixed-methods exploratory sequential study of CCM trainees, attendings, and nurse practitioners working in any of five University of Alberta affiliated medical-surgical intensive care units across Edmonton, AB, Canada (Fig. 1). We selected an exploratory sequential design, as after an initial exploratory phase, we aimed to identify variables that could be measured quantitatively in the second phase with the goal of producing a clear curricular structure.15 Prior to this study, the Department of CCM and its affiliated residency training program had no established CCUS curriculum, with the exception of informal teaching on vascular access (while on clinical service), academic half-days, and cardiology-based echocardiography rotations for trainees.

Figure 1
figure 1

Exploratory sequential design of this mixed methods study.

Full size image

This research project received ethical approval from the University of Alberta Research Ethics Board (Reference number: Pro00076306; 27 November 2017). Informed consent was obtained from all participants.

Qualitative focus groups

Following a focused literature review, a standardized interview guide (eAppendix 1; Electronic Supplementary Material [ESM]) was prepared by the authors (B.B., A.A.) who have content expertise in CCUS. A third author (V.J.D.) provided expertise in focus group and survey methodology. A brief moderator guide was created based on best practices.16,17 The lead investigator (B.B.) recruited participants through grand rounds and an invitational email with three reminders at two weekly intervals to CCM attendings, trainees, and nurse practitioners.

One of two individuals (A.A., A.G.) moderated the focus groups using the interview guide. Focus groups were limited to approximately 60 min and began with open-ended questions and gradually shifted to more close-ended questions. The target number of participants was six to eight members per group with at least one representative from each category (faculty, trainee, and nurse practitioner).16,17 We provided a ten-dollar Tim Hortons™ coffee card to participants as an incentive.

Quantitative survey

Thematic analysis of the qualitative data by two authors (B.B., V.J.D.) identified domains in which quantification could help consolidate preferences for content, educational materials, curricular implementation, portfolio development, credentialing, and assessment (eTable 1, ESM). Curricular content options were pulled from the core objectives published in the American College of Chest Physicians Statement on Competence in Critical Care Ultrasonography.10 The survey also included participant demographics. Questions were structured largely with an ordinal response (i.e., five-point Likert scale) and occasional free text format. Questions were revised iteratively over three rounds by two authors (B.B., V.J.D.) for wording and clarity. Two experts in CCUS (B.B., A.A.) reviewed the survey for content validity. The survey was piloted on two different CCM physicians not participating in the study who provided feedback on relevance, wording, clarity, and validity. We also estimated the time required for completion (~ ten minutes).

Using an online survey tool (SurveyMonkey Inc. San Mateo, CA, USA; www.surveymonkey.com), we distributed the final survey to all CCM attendings, trainees, and nurse practitioners with three reminders two weeks apart between May and June 2018 (participant survey, eAppendix 2; ESM). Those individuals who participated in focus groups were also invited to participate in the survey. A ten-dollar Tim Hortons™ coffee card was provided as an incentive.

Data analysis

Focus group analysis

All focus group discussions were audio recorded, transcribed, and analyzed with MaxQDA (Berlin, Germany). To achieve interrater reliability and trustworthiness, two investigators (B.B., V.J.D.) independently reviewed the complete transcripts. After the first reading by both authors, an established coding scheme was agreed upon and an inductive content analysis was performed to explore patterns and themes. Further, codes were characterized by frequency and by number of speakers.18 We identified anonymized quotes that were representative of individual codes. We used a constant comparative method until content saturation was achieved, as outlined by Glaser.19 Finally, we displayed the organization of our coding schema with a representative diagram.

Survey analysis

We used SPSS (IBM Corp, Armonk, NY, USA) for descriptive statistics, and for inferential statistical analysis, specifically Friedman’s test (two-way analysis of variance) to compare preference options to assure that differences were not due to chance. An investigator (B.B.) performed a thematic analysis on free text responses. To define the content of the curriculum, we required greater than 60% agreement with the statements “probably would incorporate” or “definitely would incorporate” for inclusion of content into the final CCUS curriculum.

Results

Focus group results

Three focus groups were conducted between November 2017 and March 2018 with 19 CCM practitioners (12 faculty, three fellows, and four nurse practitioners) to achieve content saturation. The proportion of participants was a near-representative sample of the larger population of CCM practitioners. eTable 2 (ESM) displays major themes and sub-themes coded by comment frequency, number of respondents, and representative quotes. Figure 2 is a visual demonstration of the sub-themes under each theme. The following section will explore key concepts within each major theme.

Figure 2
figure 2

Visual representation of number of themes and sub-themes raised during focus group.

Full size image

Theme 1: Perceived beneficial aspects of CCUS

Participants were most concerned with using CCUS to inform decision-making at the bedside in “real-time”, as an extension to the clinical exam. This emphasis on decision-making was often invoked for times where patients could not be imaged traditionally, based on severity of illness.

Theme 2: Existing limitations to learning and performing CCUS

Access to a local “champion” or “skilled mentor” was highlighted most frequently as a limitation to learning CCUS. Further, many pointed to a disparity between attendings’ and trainee’s proficiency based on generational differences in training. “It is the blind leading the blind, except for new trainees who had more training opportunities” one participant recounted. Participants also described frustration with current workshop-based teaching modules, as they lack a “period of maintenance” following “saturation.” Participants also emphasized that it was difficult to get feedback on their CCUS examination and that what was needed was a clear process for image archival that permits submission for expert review.

Theme 3: Key training priorities

Participants listed focused cardiac echocardiography as their greatest priority, with a focus on dichotomous and gross visual findings (e.g., presence or absence of fluid, normal or abnormal left ventricular function). Many also called for improved consistency in training for procedural guidance especially for “those that are trained today and for those trained in the past without ultrasound.”

Theme 4: Perceived risks or concerns in the use of CCUS

A number of participants reflected on the risk of “over-interpretation” in an abbreviated training program. One participant highlighted that the goal should be to “make people better resuscitationists, not better ultra-sonographers. The pictures you get matter, but matter less than what you do with them.” Risk was largely attributed to being “led down the wrong pathway”, rather than a direct risk of the technology itself. Finally, the absence of “quality control” was frequently mentioned as a deficiency in CCUS training; without feedback to modify future practices, participants noted they were at risk of reproducing errors.

Theme 5: Characteristics of effective instruction

Participants emphasized that learners should be aware of personal (e.g., knowledge, skill), contextual (e.g., critical illness, obesity), and technological (e.g., complexity) limitations. A number of participants also emphasized the importance of “maintaining and fostering” the skill set. Modularized learning was also preferred by many to divide the curriculum into manageable portions. Finally, hands-on mentorship at the bedside of a critically ill patient was viewed by many as essential to mastery.

Theme 6: Practices that increase likelihood of long-term training success

Many participants viewed “capture and debrief” or image archival as the ideal approach to enabling self-improvement. One participant noted that image archival could enable learning through asynchronous expert review. Participants emphasized the need for discrete periods of learning, repeated over time, to allow for “reinforcement”. This was seen to scaffold learning over time to prevent knowledge and skill attrition.

Theme 7: Assessing competency in CCUS

Image archiving and feedback was supported by many participants as key to both initial learning and ongoing assessment of CCUS competency. Portfolio development was seen as a way to ensure quantity and quality in a variety of cases.

Survey results

Participant demographics

Survey participant demographics are shown in the Table. Fifty percent (35/70) of eligible participants completed the survey; 57% were attending physicians (20/35). The most common prior CCUS experience was a weekend workshop (16/35; 46%).

Table Demographics of survey participants
Full size table

Curriculum content

Next, participants selected curricular content, which required a threshold of 60% support (as defined by responses “probably would incorporate” and “definitely would incorporate”).10 Full details are provided in the ESM (eFigs 1–4).

Safety

In a free text question about whether participants had “concerns” about the routine use of CCUS, the themes by number and percent of responders included maintenance of skill set (3/35; 9%), errors in performance (4/35; 11%), and potential medico-legal ramifications of performing CCUS (2/35; 6%).

Curriculum educational materials

Figure 3 displays the preference for learning resources; online modules (Mean = 3.3, standard deviation = 0.8) were heavily favoured above all other types of resources, a finding confirmed by Freidman’s test, which was adjusted using the Dunn-Bonferroni correction for multiple tests (X2 (3) = 30.2, P < 0.001). No other resources were mentioned by participants.

Figure 3
figure 3

Participant preferences for learning resources in a curriculum in critical care ultrasound.

Full size image

Curriculum implementation

Figure 4 displays participant preferences towards different proposed models of curriculum delivery. Preferences were distributed widely, with more than half remarking that they would “probably” or “definitely” attend bi-monthly two- to three-hour teaching sessions or high-intensity training. After statistical correction for multiple comparisons, no significant differences were found between the proposed modes of curriculum delivery.

Figure 4
figure 4

Participant preferences for delivery of a curriculum in critical care ultrasound.

Full size image

Portfolio development

We sought to understand perspectives on whether individuals would invest the additional time/effort to complete reports in return for feedback and portfolio development. Seventy-four percent (26/35) responded “yes”, 20% (7/35) answered “maybe”, and 6% (2/35) answered “no”. In a free text response to allow elaboration, five individuals (5/35, 14%) cited “time pressure” as the leading concern, which may prevent report completion.

Credentialing

In follow-up, we asked whether participants agree that a period of portfolio review should be required prior to independent performance. Nine percent (3/35) strongly agreed, 71% (24/35) agreed, whereas 14% (5/35) were neutral. Participants were also asked whether our CCUS curriculum should include a method to certify practitioners, to which 20% (7/35) strongly agreed, 60% (21/35) agreed, and 17% (6/35) were neutral.

Assessment

Given no universally accepted method to assess CCUS competency, we asked participants about the appropriateness of different methods (Fig. 5). Over three quarters of respondents felt that formal portfolio review (27/35; 77%), 1:1 proctored assessment of skill (28/35; 80%), and multiple choice based exam (27/35; 77%), were “probably” or “definitely” appropriate.

Figure 5
figure 5

Participant perspectives on appropriateness of competency-assessment methods in a critical care ultrasound curriculum.

Full size image

Integration

In this final phase, we integrated both qualitative and quantitative findings in a joint display (eTable 3, ESM). The side by side table highlights both qualitative and/or quantitative findings broken down by individual curricular elements. Based on these findings, we have developed a comprehensive and uniform framework for implementation that has been produced in collaboration with CCM practitioners.

Discussion

Our study addresses an oft neglected but key first step in curriculum development: the needs assessment. In attempting to resolve a fundamental concern with CCUS education—the gap between existing training and recommendations—we have highlighted a topic of growing importance to the needs of both practitioners and patients. Our focus group analysis uncovered major themes that describe the current state of training across a large academic institution, but also what practitioners seek from a formal curriculum. Our participants confirmed that barriers that have been previously described still exist: such as lack of skilled faculty, inability to access bedside mentorship, and perceived training time.7,8,11 Participants also provided detailed insight about less frequently discussed barriers: frustration with perceived rapid attrition of skill following one- to two-day immersive workshops, absence of a framework that sustains learning, and the inability to review acquired images.

Participants described an awareness of their cognitive and technical deficits, especially regarding unsupervised CCUS in complex, critically ill patients. Many expressed frustrations with the absence of a framework to sustain learning. Such concerns may signal a shift in CCUS education; namely practitioners—despite prior training—now need more durable learning experiences that go beyond the typical conventions of continuing medical education (i.e., one- to two-day workshops). By extension, practitioners could benefit from strategies such as site-based mentorship and longitudinal training. The quantitative portion of this research further permitted us to pinpoint details of the curriculum by examining their preferences, particularly in areas where there is still insufficient evidence.

The reported need for long-term learning suggests that CCUS champions should devote time and attention to teaching skills and preventing attrition. The main output of this research—a curriculum framework—mirrors these concerns and provides a framework to close the guideline-to-bedside gap in CCUS education. Our comprehensive efforts have included expert advice, educational theory, appreciation of frontline concerns, and a commitment to ongoing reinforcement and practical non-punitive assessment.

This study enriches the literature with a more comprehensive approach to curriculum development in CCM and could expand pre-existing recommendations. While there are studies that report the use of a needs assessment framework in CCUS, their findings are either not explicitly detailed20 or are limited to identifying a general need.21 In the latter instance, Lim et al. found universal support for point-of-care ultrasound skill development among CCM faculty and trainees, low levels of comfort with basic machine use, and a great interest in learning more about basic critical care echocardiography.21 We agree that a basic assessment is valuable to trigger the need of new curricula, but also wish to highlight how a more comprehensive process can play a greater role in shaping the curriculum itself. In addition, while surveys in both North America and Europe mirror our participants concerns, it is uncertain to what extent the more systemic problems (i.e., absence of trained faculty) have been addressed.7,8,11 Our data also bolsters previous training recommendations, such as the need for supervised practice and portfolio development.5,6,10 It also integrates newer assets such as iterative review of safety principles (e.g., error prevention, communication), spiral review (i.e., continuous review of content over time), and site-based mentorship (i.e., “hub-and-spoke” delivery) across several hospitals. Through this work, we have produced a roadmap to enable CCUS to reach its potential. This framework can be scaled down to single-centre institutions or scaled up as needed for larger organizations in Canada and even internationally.

Our approach to curriculum development is evidence-based and comprehensive.14,22 Qualitative analysis of the focus groups allowed us to unearth perspectives and ideas that might have been missed if we had only performed a survey-based approach. The process of needs assessment also identified logistical problems (e.g., machine quality concerns) and enhanced institutional support. This is because, previously, there was little local data with which to collectively guide CCUS. Our focus groups, which were largely made up of faculty (12/19: 63%) and nurse practitioners (4/19; 21%) provided a rich, albeit sometimes contrary, perspective. This not only encouraged candid dialogue but allowed participants to augment their views in real-time.

This study has limitations. First, it is a targeted needs assessment of a relatively small number of CCM practitioners from a single Canadian city (albeit across multiple hospitals). Systemic factors, including institutional culture (e.g., faculty participation in continuing medical education), may limit generalizability. While comparisons between disciplines and stage of learner would be interesting, our limited sample size prohibits meaningful subgroup analysis. Although we collected some validity evidence for our survey with piloting and content review, we did not do a formal survey validation before using it. Finally, our response rate of 50% (35/70) was lower than expected, though consistent with published response rates in physician surveys (54%).23

Future directions

Our next steps are to implement our curriculum framework and then future research will need to evaluate the effectiveness in addressing the current gaps in CCUS education to close the gap in achieving universal basic fluency.

Conclusion

Critical care ultrasound is now considered an essential competency for graduating CCM physicians. While experts recommend what should be delivered, there had been little engagement of those expected to attain and maintain CCUS competence. Our needs assessment offers a roadmap to reconcile expert recommendations with the concerns, priorities, and preferences of frontline clinical practitioners. Finally, we propose a new curriculum framework to achieve universal basic fluency. This investigation provides new insights to those educators looking to develop their own curricula, including concerns regarding attrition of skill following workshops and absence of initiatives to sustain learning. Future research will focus on implementing and evaluating this new curriculum framework to measure its impact on the described needs of frontline practitioners.