Anesthesiologists’ learning curves for bedside qualitative ultrasound assessment of gastric content: a cohort study
- First Online:
- Cite this article as:
- Arzola, C., Carvalho, J.C.A., Cubillos, J. et al. Can J Anesth/J Can Anesth (2013) 60: 771. doi:10.1007/s12630-013-9974-y
- 1k Views
Focused assessment of the gastric antrum by ultrasound is a feasible tool to evaluate the quality of the stomach content. We aimed to determine the amount of training an anesthesiologist would need to achieve competence in the bedside ultrasound technique for qualitative assessment of gastric content.
Six anesthesiologists underwent a teaching intervention followed by a formative assessment; then learning curves were constructed. Participants received didactic teaching (reading material, picture library, and lecture) and an interactive hands-on workshop on live models directed by an expert sonographer. The participants were instructed on how to perform a systematic qualitative assessment to diagnose one of three distinct categories of gastric content (empty, clear fluid, solid) in healthy volunteers. Individual learning curves were constructed using the cumulative sum method, and competence was defined as a 90% success rate in a series of ultrasound examinations. A predictive model was further developed based on the entire cohort performance to determine the number of cases required to achieve a 95% success rate.
Each anesthesiologist performed 30 ultrasound examinations (a total of 180 assessments), and three of the six participants achieved competence. The average number of cases required to achieve 90% and 95% success rates was estimated to be 24 and 33, respectively.
With appropriate training and supervision, it is estimated that anesthesiologists will achieve a 95% success rate in bedside qualitative ultrasound assessment after performing approximately 33 examinations.
Les courbes d’apprentissage des anesthésiologistes pour l’évaluation du contenu gastrique par échographie qualitative au chevet: une étude de cohorte
L’évaluation ciblée de l’antre pylorique par échographie est un outil utile pour estimer la qualité du contenu de l’estomac. Notre objectif était de déterminer la quantité de formation nécessaire à un anesthésiologiste afin d’acquérir des compétences en échographie au chevet pour l’évaluation qualitative du contenu gastrique.
Six anesthésiologistes ont suivi une formation, puis une évaluation formative; des courbes d’apprentissage ont ensuite été élaborées. Les participants ont reçu un enseignement didactique (matériel de lecture, bibliothèque d’images, et cours) et suivi un atelier pratique interactif sur des modèles vivants dirigé par un échographiste expert. On a enseigné aux participants la façon de réaliser une évaluation qualitative systématique afin de diagnostiquer l’une de trois catégories distinctes de contenu gastrique (vide, liquide clair, solide) chez des volontaires sains. Les courbes d’apprentissage individuelles ont été construites sur la base de la méthode des sommes cumulées. La compétence était définie en tant que taux de réussite de 90 % dans une série d’examens échographiques. De plus, un modèle de prévision a été créé sur la base de la performance de la cohorte entière.
Chaque anesthésiologiste a réalisé 30 examens échographiques (soit un total de 180 évaluations), et trois des six participants ont atteint le niveau de compétence prédéfini. Le nombre moyen de cas requis pour atteindre des taux de réussite de 90 % et 95 % a été estimé à 24 et 33, respectivement.
Avec une formation et une supervision adaptées, on estime que les anesthésiologistes atteindront un taux de réussite de 95 % des évaluations échographiques qualitatives au chevet après avoir réalisé environ 33 examens.
Pulmonary aspiration of gastric content is considered a major complication in anesthesia practice with resulting morbidity and mortality.1-4 Certain factors have been linked to the severity of patient outcomes, including the volume, nature, and acidity of the aspirate.5-8 The preoperative assessment of the risk of pulmonary aspiration relies essentially on the patient’s history, and the clinical management typically adheres to fasting recommendations of current guidelines.9 The ultimate assessment of the nature and volume of gastric content at bedside remains inaccessible to the anesthesiologist.
Recent studies have shown that bedside ultrasonography can provide reliable information about the nature (clear fluid, solid, or none) and volume of gastric content.10-14 In these studies, either a certified sonographer or a single anesthesiologist with no specified previous training performed all the examinations.11-13,15-17 The process of knowledge translation demands further research to confirm generalizability of these findings. The amount of training required to achieve competence in the performance of gastric ultrasonography as a bedside clinical tool remains unknown. The aim of this study was to determine the amount of training an anesthesiologist would need to achieve competence in the bedside ultrasound technique for qualitative assessment of gastric content in healthy volunteers.
Following approval by the Research Ethics Board (ref: 11-0237-E) of Mount Sinai Hospital (Toronto, Canada) on October 14, 2011, we conducted this observational cohort study of a group of anesthesiologists learning qualitative ultrasound assessment of gastric content in healthy volunteers along with the reference standard of an expert sonographer. The study was carried out from October 2011 to December 2011. Written informed consent was obtained from all participants (anesthesiologists, volunteers, and expert sonographer). We followed the guidelines for reporting and the applicable checklist from the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) Statement.18
Eligible participants were advanced level trainees (anesthesia fellows during one year of subspecialty training) or staff anesthesiologists. Participants had previous experience in ultrasound-assisted/guided procedures, but not in gastric ultrasound.
Inclusion criteria for volunteers were: age 18 yr or older, weight 50-120 kg, and height 150 cm or greater. Volunteers with a pre-existing abnormal anatomy of the upper gastrointestinal tract (previous esophageal, gastric, or upper abdomen surgery and hiatus hernia) were excluded. A certified sonographer experienced in diagnostic abdominal ultrasonography with more than 500 examinations of gastric content was considered the reference standard for the ultrasonographic assessments. The cohort of anesthesiologists underwent a teaching intervention which was followed by a formative assessment process in order to construct learning curves.
Initially, the cohort of anesthesiologists received didacticteaching in the form of reading material, educational video/picture library, and a 30-min lecture on the topic. The reading material included research studies11-14 and one educational article with a description of the technique.19 Subsequently, the teaching proceeded with an interactive three-hour session consisting of a hands-on workshop and a live demonstration performed by the expert sonographer. Five volunteers were scanned after fasting for at least eight hours, and they were later scanned after ingesting either clear fluids (300 mL of apple juice) or a standardized solid meal (muffin and apple juice). Each anesthesiologist received one-on-one feedback on ten exams that included every prandial status (fasted, clear fluids, and solid meal). During the workshop, the anesthesiologists were instructed on how to perform a systematic ultrasound assessment in order to make a qualitative diagnosis of three distinct gastric content categories (empty, clear fluid, solid).
The formative assessment process consisted of a series of ultrasound examinations performed by each anesthesiologist followed by confirmation and feedback from the expert sonographer. We scheduled six sessions with five volunteers per session, with 30 separate sets of data per anesthesiologist. For each evaluation session, five study volunteers were randomly allocated to one prandial status similar to that previously described in the hands-on workshop (eight-hour fasting, clear fluids, and solid meal). Randomization allocation was in two blocks of 15. Each block of 15 included the three gastric content categories: five empty, five clear fluids, and five solid. The allocation concealment comprised of sequentially numbered and sealed opaque envelopes prepared by an independent research assistant. At each session, the ultrasound examinations were started five minutes after the ingestion of liquid or solid, and the assessment session was paused every 15 min to provide a top-up of clear fluids (100 mL) to the fluid group, which was also blinded. The purpose of the top-up was to attempt to maintain equal examination conditions due to possible rapid gastric emptying of clear fluids in a fasted volunteer. The expert sonographer and the anesthesiologists were blinded to the volunteer prandial status, and they examined all five volunteers in a random sequence. Additionally, each of the six sessions was scheduled one week apart (six-week assessment period), and each volunteer was scanned only once per session by each anesthesiologist. The prandial status was disclosed only when the examination of the five volunteers was completed by all anesthesiologists and the sonographer. The expert sonographer then conducted a 50-min feedback session, including review of images and discussion and reinforcement of the technical aspects. In order to maintain uniform learning progress for the entire group, we did not allow participants to practice the ultrasound technique between sessions; however, they were encouraged to review the didactic teaching material.
The aim of the qualitative ultrasound examination was to diagnose three distinct gastric content categories (empty, clear fluid, or solid). Success was defined as a correct diagnosis, which was recorded over time. Although the diagnosis from the expert sonographer was the reference standard by study design, we also followed her performance, as she was also blinded to the volunteers’ prandial status. In this way, we were able to show the construct validity of the technique. We aimed to construct learning curves for a series of ultrasound examinations through the cumulative sum method (CUSUM).20
The primary outcome was the number of ultrasound examinations (cases) an anesthesiologist required to achieve competence in the ultrasonography qualitative assessment of gastric content. Competence was defined as 90% success rate in a series of ultrasound examinations.20-22 The secondary outcomes included the overall success rate of correct ultrasound diagnoses, the specific success rate per gastric content category, and characteristics of the series of ultrasound examinations.
In order to estimate the primary outcome for each anesthesiologist, individual learning curves were constructed using the CUSUM graphical method (Appendix 1). The CUSUM graphs display the cumulative differences plotted in sequence, allowing detection of deviations from a predetermined standard.20-26 A negative trend of the individual CUSUM line indicates success, whereas a positive trend indicates failure. We considered the anesthesiologist competent at a 10% failure rate if the graph passed through two decision lines from above. In this way, early failures did not penalize the anesthesiologist for later increases in accuracy.25,26
We calculated a minimum sample size of 13-18 consecutive cases as necessary to determine competence at an acceptable failure rate of 10% and an unacceptable failure rate of 30%, respectively (Appendix 1). We increased the sample to 30 consecutive cases per anesthesiologist (twice the average of calculated cases) to allow an adequate assessment process.
Furthermore, we used the entire cohort data in the construction of a predicted learning curve based on Bush and Mosteller’s mathematical learning model, which estimates the average number of cases to achieve a 95% success rate (Appendix 2).23,27 In this way, we were able to determine an average CUSUM score and construct a learning curve for the average of all participants, allowing interpretations for the entire cohort rather than for each individual alone.
The assessments were also compared with respect to a correct or incorrect diagnosis. We examined the effect of the gastric content category on the correct ultrasound diagnosis. For the comparisons, we used the generalized regression models with a generalized estimating equation approach accounting for the correlated data. Further evaluation of the effect of the gastric content category and the duration of the ultrasound examination was conducted using multiple logistic regression models for correlated data.
Descriptive statistical methods were used to describe the study population. The statistical analyses were performed using SAS® 9.2 (SAS Institute Inc., Cary, NC, USA), R 10.2 (http://www.r-project.org/), and STATA® for Macintosh, Release 12.1 (StataCorp, College Station, TX, USA). A two-sided significance level of < 0.05 was used without multiple comparison adjustment.
The cohort of anesthesiologists consisted of four anesthesia fellows and two anesthesia staff. All participants completed the teaching and evaluation sessions. The group of healthy male volunteers had a mean (standard deviation = SD) age of 25 (2) yr, weight of 71 (12) kg, height of 173 (4) cm, and body mass index of 25 (3) kg·m−2.
Incidence of correct (Success) and incorrect diagnosis (Failure) according to the gastric content category and the duration of the ultrasound examination
n = 142
n = 38
Gastric content category n (%)
2.3 (1.4 to 3.1)*
3.9 (2.1 to 5.7)*
Diagnostic failure among the three gastric content categories and duration of the ultrasound examination
Gastric content category
n = 60
n = 60
n = 60
Failure n (%)
3.3 (2.1 to 4.5)†
1.3 (0.3 to 2.3)†‡
3.3 (1.8 to 4.8)‡
Multivariable analysis: odds of success according to gastric content categories
Solid vs Empty*
(1.04 to 6.6)
Solid vs Clear Fluid
(0.6 to 4.8)
Clear Fluid vs Empty
(0.6 to 3.8)
(0.8 to 1.1)
This study determined the learning curves for the qualitative ultrasonography assessment of gastric contents performed by a cohort of anesthesiologists. Under the study conditions, CUSUM analysis revealed that 24 consecutive ultrasound examinations were required to achieve competence with a 90% success rate, and at least 33 examinations would be required for a 95% success rate based on a predicted model. The fact that the expert sonographer correctly diagnosed the gastric content in every volunteer confirms the construct validity of the technique. Nevertheless, according to CUSUM analysis, only three of six novice participants achieved competence in the series of 30 examinations. This may indicate a medium level of complexity in learning the technical and cognitive skills necessary to perform this ultrasound diagnostic procedure.
The CUSUM method has been used in many studies investigating the acquisition of competence or assessment of performance in anesthesia technical skills,22,25,28-31 and it has been used more recently in ultrasound-guided procedures.21,23,24,26,32 It can be used to show proficiency in a newly learned technical skill or measure quality once a technical skill has been mastered.33 In the context of medical education, it is crucial to understand CUSUM as a statistical method that looks at the outcome rather than the process of performing procedural skills.20,28 Although we focused on the formative assessment (training, feedback, discussion) aspect in our study, achieving and declaring competence still reflects the goal of summative assessment.34-36 Defining competence is a complex task. Acceptable rates of success can be determined by institutional rates or expert consensus, but they also depend on the definition of success.20,30 Currently, there is no standard for this new diagnostic skill in the anesthesia setting. In some studies on newly acquired procedural technical skills in anesthesia, 90% success rates have been used based on previous training studies21 or departmental consensus using a modified Delphi approach.22 Therefore, for this study, we decided to define competence as a 90% success rate as determined by consensus of opinion among the investigators who had performed previous research on ultrasound assessment of gastric content.
If we had followed the method of pooled cumulative success rate described by Kopacz et al.,37 all participants could have been assessed by their actual overall success rate, which was greater than 70% for all six anesthesiologists and corresponds to the predefined cut-off point of 30% unacceptable failure rate. Nevertheless, the CUSUM method tends to benefit consistency over a sequence of cases, and its value comes from providing statistical inference while testing the failure rate against a minimum standard. Furthermore, if we assume that all anesthesiologists had the same baseline experience, we can use the summated changing success rates to develop a learning curve for the average of all participants until a certain target rate is achieved in this outcome-based task. In this way, the predicted model facilitates making interpretations for a population rather than for each individual alone, providing potential application for curriculum development, practice rotations, and mentor evaluations.38
We planned a study protocol with very distinct gastric content categories to enhance the early stages of learning. Among the three gastric content categories, study results suggest that “solid” content appears to be more easily diagnosed by novice sonographers, followed by “clear fluid”, and then the “empty” category. Additionally, the “clear fluid” is more readily appreciated. We managed to optimize the clear fluid content group in order to avoid the possible confounding factor of rapid gastric emptying. The antrum filled with clear fluid is more obvious to the observer, appearing round and filled with hypoechoic/anechoic content (Fig. 1B). Nevertheless, the “solid” content group appeared to have been diagnosed more easily than both the “clear fluid” and the “empty” groups. This may be related to the relatively larger size of the antrum and the characteristic hyperechoic mucosal-air interface in cases of recent ingestion of solid food that depicts a “frosted glass” appearance, impairing visualization of deeper structures (Fig. 1C).19 Further research is warranted to determine the particular challenges and difficulty in the diagnosis of various gastric contents.
Our study has some limitations. First, the quantitative dimension (volume assessment) was beyond the scope of this study. This aspect of the ultrasound assessment is not required in the diagnosis of “empty stomach”. Nevertheless, if the gastric sonogram should show clear fluid content, a quantitative volume assessment can help differentiate a low-volume status (similar to baseline physiologic gastric content) from a high-volume status (higher than baseline gastric volume).11-13 We concentrated instead on using a focused qualitative exam as an approach to a screening test. Second, although we assessed quite distinct categories of gastric content and performed the ultrasound assessments at fairly constant time points, scanning conditions for the individual participants could have varied during the course of the session. Third, our five volunteers were healthy young males with a normal body mass index; while we believe that this was essential for reliable and valid training, results may not be fully applicable to other populations. Although we had a limited number of volunteers and bias was a concern, we planned to minimize such bias by carrying out each of the six sessions one week apart (six-week assessment period), and each volunteer was scanned only once per session by each anesthesiologist. Fourth, some of the predefined statistical variables of CUSUM in the study design may have limited participants’ achievement of competence. Specifically, the learning curves were defined by the level to declare competence, the number of participants, and the individual attempts per participant. Finally, this cohort of anesthesiologists had previous experience in other applications of ultrasound for diagnostic and interventional procedures in anesthesia. This baseline condition could have facilitated the development of their learning curves and could also make conclusions more suitable for this particular population.
The use of ultrasound by anesthesiologists as a perioperative point-of-care resource has dramatically increased over the last ten years, and many current best practices entail ultrasound to guide clinical decision-making and procedures. A growing body of evidence shows the benefits of this change in practice.39 In this context, bedside ultrasound could potentially become a clinically useful noninvasive tool for accurate determination of gastric content and volume. It could help to establish the risk of perioperative aspiration more precisely, both in a given individual and in different patient populations.9,12,14,40 Practice guidelines are subject to revision as warranted by the evolution of medical knowledge, technology, and practice.9,41 Factors such as current literature, expert and practitioner opinion, open forum commentary, and clinical feasibility data support these recommendations. Presently, it may be early and premature to anticipate how this new technology could affect practice, although we believe it is a promising clinical skill to pursue.
In conclusion, we have described the feasibility of appropriate training of anesthesiologists to reach a level of competence in performing bedside qualitative ultrasound assessment of gastric content. Individual learning curves and a predictive model suggest that 24 and 33 are the average numbers of cases required to achieve 90% and 95% success rates, respectively.
The authors thank Liisa Davis B.Sc., Sonographer, Regional Anesthesia Research Group, Toronto Western Hospital, University Health Network, Toronto, Ontario, Canada, for her invaluable technical assistance in training and subject examinations. The authors also thank Kristi Downey MSc, Perinatal Research Coordinator, Department of Anesthesia and Pain Management, Mount Sinai Hospital, for her invaluable contribution to all stages of this project.
This work was supported by departmental resources.
Conflict of interest
Dr. Anahi Perlas is Associate Editor of the journal, Regional Anesthesia and Pain Medicine. Dr. Anahi Perlas receives support for academic time from a University of Toronto, Department of Anesthesia Merit Award 2011-2013. None of the other authors have financial or personal relationships or affiliations that could influence (or bias) this research.