Introduction

Eye-Tracking is an objective tool, which delivers insights into gaze behavior of the operator during medical procedures [1,2,3]. With this innovative technology the visual focus of a healthcare provider during performance of medical tasks can be determined [4,5,6]. Although gaze behavior cannot be equated with cognition, it enables a better understanding of the thought processes of operators with various levels of expertise. Thus, differences in gaze patterns between experts and novices have been reported in several studies. Experts mainly remain longer in predefined areas of interest and have a clear focus to their gaze [7,8,9]. Moreover, studying an expert’s gaze behavior may even play a role in medical education for better understanding an expert’s approach and method, especially for challenging tasks (e.g., ultrasonography in critical care medicine).

Due to the technical progress of portable ultrasound devices and appropriate education of nonradiologists and noncardiologists in sonography, point-of-care ultrasound (POCUS) is increasingly used in critical care medicine [10]. POCUS provides real-time diagnosis and supports physicians in decision making to develop treatment strategies [11]. As clinicians with insufficient knowledge may harm patients with inaccurate diagnosis, training to acquire adequate competence is essential. However, the optimal training (e.g., course design, practical training) for becoming competent and the best way in which experts can deliver support for novices during POCUS is unknown.

A handful studies investigated the technology of eye-tracking in medical education for training, assessment as well as a feedback tool [4, 8, 12, 13]. Several studies reported that gaze trained novices demonstrated patterns closer to experts [12, 14,15,16]. Furthermore, in the educational setting eye-tracking can even be used as a feedback tool. This allows trainees to focus on the area, where training and improvement is required [12]. Due to these findings, insights into gaze behavior between experts and novices may be a useful step for teaching POCUS. Nevertheless, eye-tracking has not been studied during echocardiography.

Therefore, the aims of this study were to investigate technical feasibility as well as usability of eye-tracking during simulation-based echocardiography and to evaluate differences of gaze patterns between experts and non-experts.

Methods

Study design

This feasibility study was approved by the data protection committee of the Medical University of Vienna, Austria (05/2021) and was performed according to the Declaration of Helsinki guidelines regarding research on human subjects. Due to the manikin-based simulation character of this study the ethics board gave this study an exempt status. This study was performed between September 2021 and March 2022 at the Medical University of Vienna.

Participants

In this study, nine clinicians with expertise on critical care echocardiography and six residents with limited expertise were recruited to voluntarily participate in the study. Informed written consent was obtained from all participants before beginning. Experts were specialists from the Department of Internal Medicine as well as from the Department of Anaesthesia, Intensive Care Medicine and Pain Medicine with diploma and regular use of focus assessed transthoracic echocardiography (FATE) for more than two years. As non-experts, six residents of the Department of Anaesthesia, Intensive Care and Pain Medicine with theoretical courses, but low practical expertise on echocardiography, were included. All residents participated in an 1 h online repetition on the basics of echocardiography prior to the study on the basics of echocardiography.

Experimental setup

Prior to the study initiation, six pre-installed medical cases of the simulator U/S MENTOR™ (3D Systems, Airport City, Israel) were selected by the study team. For each, scenario participants were given details about history and clinical symptoms of the patient (supplementary data 1). To ensure reproducibility and to offer optimal conditions for the interpretation of ultrasound images, a standardized set-up for simulator, ultrasound monitor, chair and ambient lighting was provided and checked before each scenario. Participants were equipped with mobile Tobii Pro 2.0 eye-tracking glasses (Tobii Pro, Stockholm, Sweden). Calibration of the eye-tracking glasses was performed according to the manufacturer’s instruction before each scenario. During echocardiography, the participants described orally at which anatomical structure they were looking at. Audio and eye movements at a rate of 50 Hz were recorded with the eye-tracking glasses. For analysis, the Tobii Pro Glasses Analyzer Software (Tobii Pro, Stockholm, Sweden) was used. The areas of interests (AOIs) were determined by the study team for each of the standard FATE views (parasternal long axis, parasternal short axis, apical and subxiphoidal) for every scenario in accordance with the fixation patterns observed from the first three experts. The sequence of the scenarios was randomized using latin squares to account for habituation and learning effects.

Outcome measures

Feasibility was defined as the accordance of the orally described area of visual attention with the area of attention marked by the eye-tracking glasses (n = 12). For differences between experts (n = 6) and non-experts (n = 6), the relative dwell time (focus on a specific AOI) as well as the absolute time to first fixation in the case specific AOI, and total duration of echocardiography were assessed. The usability of the eye-tracking glasses was evaluated by all participants (n = 15) after the last scenario using a questionnaire with 11 items about comfort and distraction (5-point Likert scale for each item) which was also used in a previous feasibility study [17]. Baseline characteristics included age, gender, medical background, years of professional experience as well as years of experience in echocardiography.

Sample size calculation

For this feasibility study, sample size was based on previous work by Borg et al., who analyzed the feasibility of eye-tracking glasses during ultrasound-guided regional anaesthesia with 5 experts and 5 non-experts [6]. For determination of the case specific AOIs it was suggested that in total 3 experts were sufficient.

Statistical analysis

Descriptive statistical analysis was performed using SPSS Version 27 (IBM, Armonk, NY, USA), while feasibility, usability, and the comparison between experts and non-experts was analyzed using SAS 9.4 (SAS Institute, Cary, NC, USA, 2016). Demographic data are presented as median (minimum and maximum) due to an asymmetric distribution of some variables. The accordance of the verbally stated area of visual attention with the marked area are reported as percentage of the correctly stated areas of the total number of verbalized areas of visual attention. Relative dwell time, as well as first time of fixation and duration of echocardiography, are described as median (minimum and maximum) for each group (experts; non-experts). Differences between these groups were estimated using a linear mixed model with fixed effects group, case, and period, and a random proband effect to adjust for period effects and dependence within probands. The conditional model residuals were successfully checked for approximate normal distribution and for outliers and influential observations. The conditional R-squared measure is calculated using the function r.squaredGLMM in the R package ‘MuMln’ Version 4.3.0 [18]. Time to first fixation was log-transformed to stabilize the residual distribution (group differences are back-transformed to result in a relative group difference); an additional random effect for view within proband was added. Usability of eye-tracking glasses are reported as mean for each group.

Results

Participants’ characteristics

Demographic data of the participants are depicted in Table 1.

Table 1 Demographic data
Full size table

Technical feasibility

During the entire duration of all 72 recordings, the area of attention was marked by the eye-tracking glasses. In total 1366 areas were orally explained. In 96% of these, the orally described area was in accordance with the marked area of attention.

Differences between experts and non-experts

Differences between experts and non-experts are depicted in Table 2. Although not statistically significant, experts had a longer relative dwell time in the defined AOI than non-experts (raw data across all cases: 50.6% versus 38.4%; adjusted difference: 9.4%-points, 95% confidence interval (CI): -0.9 to + 19.6, p = 0.072, R2 0.51). Furthermore, experts performed a faster ultrasound examination (median raw data across all cases:138 s versus 227 s; adjusted difference: 68 s, 95% CI: -5 to + 142, p = 0.068, R2 0.72). The median time to first fixation in the AOI (supplementary data 2) was significantly lower in the expert group compared to the non-expert group (median raw data across all cases and views: 5 s versus 10 s; adjusted relative difference: 45% reduced fixation time, 95% CI: 68% to 5%, p = 0.033, R2 0.18). Differences of gaze patterns between experts and non-experts are demonstrated in Fig. 1.

Table 2 Differences between experts (n = 6) and non-experts (n = 6)
Full size table
Fig. 1
figure 1

Heatmap of gaze patterns (case 2 pulmonary embolism) between experts (n = 6) and non-experts (n = 6). Intensity of redness correlates with the dwell time in the area

Full size image

Usability of eye-tracking glasses

At the end of all 6 scenarios each participant evaluated the subjective experience with the eye-tracking glasses (Table 3). 13 of 15 participants denied that eye-tracking glasses distracted them from performing echocardiography. Most participants disagreed with discomfort due to eye-tracking glasses. Whereas all non-experts would wear eye-tracking glasses during a real scenario, only 2 of 9 experts considered this option.

Table 3 Usability of eye-tracking glasses
Full size table

Discussion

To our knowledge, this is the first study evaluating eye-tracking during simulation-based echocardiography. The findings of this pilot study demonstrated that eye-tracking during transthoracic echocardiography was technically feasible, relatively comfortable for the performer, and suggested that non-experts and experts have different gaze patterns.

Eye-tracking and assessment of gaze behavior holds a great potential for educational purposes, as well as for understanding and quantifying the development of expertise in POCUS [6, 19]. In previous studies, eye-tracking was investigated while participants were looking at prepared sonography images (e.g., trauma sonography, regional anaesthesia, coronary angiogram) and demonstrated that analysis of gaze patterns is even feasible in case of moving POCUS examinations [6, 19,20,21]. Due to the development of portable ultrasound machines, POCUS is not only reserved to specialists (e.g. cardiologists) anymore. POCUS gains in importance in emergency medicine, even in out-of-hospital scenarios as it may be useful for urgent triage, guidance of procedural interventions, evaluation of clinical symptoms, therapeutic monitoring as well as during cardiopulmonary resuscitation [22, 23]. Although, training in POCUS is recommended for medical students as well as for residents and specialists, the most effective training method for POCUS is unknown. Interpretation of gaze behavior during POCUS examinations, which was demonstrated in this trial to be technically feasible, opens new horizons for educators and trainees as eye-tracking can be used for training, assessment as well as a feedback tool to improve individual’s learning curve.

In this pilot study experts and non-experts had different gaze patterns, although only the duration to first fixation in the AOI was statistically significant (5 s versus 10 s, p = 0.033). These findings are comparable with previous studies. Differences in gaze fixation behavior between experts and non-experts were detectable for image interpretation of focused assessment with sonography for trauma, ultrasound-guided regional anaesthesia as well as for coronary angiogram [6, 19,20,21]. Several studies demonstrated that experts spend less time in identifying anatomy and thus pathologic findings were focused more quickly [6, 24]. In the field of surgery, novices were trained with gaze patterns of experts leading to faster and more accurate performance [8]. Analyzing gaze paths recordings of experts could even be useful for novices trained in POCUS to visualize regions of interests in cases with different pathologies.

This pilot study carries some limitations. First, due to the small sample size interpretation and generalization of the findings of this study should be done with caution. The 95% confidence interval for the percentage of correctly identified AOIs (92.3% to 98.9%) is fully above 90%, which was the acceptable lower bound defined in the study protocol for establishing feasibility. A formal sample size calculation has not been performed for this study, which could have led to insufficient data. However, the low width of the confidence interval demonstrates sufficient power for this feasibility study, though only in a post-hoc manner. Moreover, the non-experts in this study had a median experience of echocardiography of 5 (0.5–7) years and thus cannot be classified as complete novices.

Conclusion

In conclusion, the findings of this study support the feasibility of eye-tracking during POCUS examinations and suggest that interpretation of gaze patterns has the potential to become a tool for clinical education. For trainees, analysis of gaze behavior may facilitate the evaluation of learning and may provide valuable feedback about ultrasound performance. Future studies are recommended to explore this technology and its relevance for educational purposes of POCUS.