In the event of an anesthesia machine malfunction, anesthesiologists must troubleshoot and, ideally, rectify the problem. Catchpole et al. 1 reviewed 12,606 reported incidents from the United Kingdom National Reporting and Learning Service and found a 13% incidence of problems involving medical devices associated with anesthesia patient care. Webb et al.2 reported a 9% incidence of “pure equipment failure” in their review of 2,000 incidents reported to the Australian Incident Monitoring Study. Sixty percent of these 177 cases were related directly to anesthetic equipment failure, and 55% were deemed to have “potentially life-threatening consequences”.2

As equipment faults and failures can occur, it is intuitively important to perform a preoperative anesthesia machine check, yet machine faults are still missed regardless of anesthesia experience. Buffington et al.3 asked 179 volunteers to check an anesthesia machine preset with five faults; only 3.4% found all five errors, and an astounding 7.3% detected no faults at all. Larson et al.4 found an inverse relationship between the number of years of anesthesia experience and the number of preset anesthesia machine faults detected by those providers.

The introduction of anesthesia machine checklists by the United States Food and Drug Administration (FDA) and the Association of Anaesthetists of Great Britain and Ireland over the last two decades has not enhanced anesthesiologists’ recognition of anesthesia equipment faults.5,6 In addition, few studies have been conducted to evaluate how best to teach trainees to perform a proper anesthesia machine check.7,8

The purpose of this study was to determine if 1) an experiential training session would improve junior (postgraduate year 1 or PGY-1) anesthesiology residents’ ability to perform a thorough machine check and correctly identify preset faults beyond the level of final year (PGY-5) residents who had received only a didactic training session; and 2) to determine if the junior residents’ ability would be retained over time until their senior (PGY-3 to PGY-5) year.

Methods

This study was approved by The Ottawa Hospital Research Ethics Board. Written informed consent was obtained from all participants prior to their participation in the study. In 2005, an experiential training session was designed to teach residents how to perform an anesthesia machine check and detect anesthesia machine faults. This experiential training session was introduced into the residency training program at the PGY-1 level (the residents received the training at the beginning of their two-month anesthesia rotation, which occurs at the end of their first year of residency). The control group included PGY-5 residents who received a traditional didactic lecture on the anesthesia machine as part of the Anesthesia Equipment core program series, which is delivered on a three-year cycle (thus, the control residents would have received this lecture as a PGY-2, 3, or 4). Data were collected prospectively from 2005 to 2008 with each incoming group of PGY-1 residents and graduating PGY-5 residents. When the original group of PGY-1 residents became PGY-5 residents in 2009, they were invited to return for a retention test.

At the end of their first year of training, all simulation residents were also given a didactic lecture on the anesthesia machine; the lecture was identical to the lecture given to control residents, and it was delivered by the same faculty as part of the Anesthesia Equipment core program. This lecture was followed by an experiential hands-on training session on the anesthesia machine, which consisted of a brief presentation on the anesthesia machine followed by a demonstration of a full machine check using a 45-item modified FDA checklist9 (Appendix 1). The FDA checklist was customized to fit the GE Datex-Ohmeda Aestiva 5 gas machine (GE Healthcare, Madison, WI, USA) using the checkout procedures recommended in the GE Datex-Ohmeda manuals.Footnote 1 This modified checklist was comparable with the GE Datex-Ohmeda Aestiva checklist on the University of Florida’s Virtual Anesthesia Machine website9 (http://vam.anest.ufl.edu/). The 120-min experiential training session started with a 30-min overview on the operation of the gas machine with diagrams and demonstrations on an actual gas machine. The next 60 min were spent reviewing each step of the checkout procedure. During this time, the specific faults were not shown, but problems that could be encountered at each step of the checkout were discussed. The residents were encouraged to ask questions and interact. In the final 30 min, the checkout was repeated in an orderly fashion to show how the machine flowed and came together logically. There was no debriefing or evaluation at this initial session; however, the residents were given the opportunity to perform a machine check under direct supervision with feedback.

Three weeks later (post-test), the simulation residents returned for assessment. The residents were asked to detect as many machine faults as possible (the number of preset faults was not mentioned), and the number of faults detected was recorded. The ten preset faults are listed in Appendix 2. All machine faults were corrected, and the residents were then asked to perform a complete anesthesia machine check. Each resident’s score on the 45-item checklist was recorded, as was the time each resident took to perform a complete machine check. The missed faults were revealed to the resident after the test session, and the importance of detecting the faults was emphasized. The resident was also shown the steps that were missed in the checklist and how to perform them. The resident’s scores for both the machine faults and the machine check were evaluated in real time by a single attending anesthesiologist. Residents were invited to repeat both tests in their senior year (PGY-3 or higher) of residency (retention test).

The control residents were asked to find the same ten preset faults and complete a machine check at the end of their training (control test). The number of faults detected, the time taken for the machine check, and the score on the 45-item modified checklist were scored in real time by a single attending anesthesiologist.

Statistics

Simulation post-test, simulation retention test, and control tests were compared using an analysis of variance (ANOVA) for parametric data with a Tukey post hoc analysis for significant findings. Non-parametric data were compared using the Kruskal-Wallis test. Differences were considered significant at P < 0.05.

Results

Forty-two PGY-1 residents were approached and 37 participated in the study. Forty-three PGY-5 residents were approached and 27 participated in the study. Twenty-one residents of the original 37 PGY-1 residents who participated returned for the retention test. Table 1 describes the groups, the time of training during their residency, and the type of training they received.

Table 1 Description of study groups and training received
Full size table

Junior simulation residents identified more preset machine faults than senior control residents (P < 0.001) in their post-test (Table 2). Junior simulation residents also achieved significantly higher checklist scores than senior control residents with no experiential training (P < 0.001). Twenty-one of the simulation residents repeated the study in their senior year, and they continued to achieve higher checklist scores and identify more machine faults than senior control residents who did not receive experiential training (P < 0.001) (Figure; Table 3). The senior control residents took less time to check the machine than either the junior simulation or senior simulation residents (P < 0.001).

Table 2 Summary of results for senior control, junior simulation, and senior simulation residents
Full size table
Figure
figure 1

Mean checklist score and number of machine faults detected expressed as a percentage (standard deviation) for senior control and junior and senior simulation residents

Full size image
Table 3 Summary of checklist items performed correctly and machine faults detected correctly in each group
Full size table

Discussion

This study showed that junior anesthesiology residents who received a focused experiential training session on the pre-use check of the anesthesia machine were significantly better able than senior anesthesiology residents without experiential training to detect preset machine faults and perform a thorough machine check.

Olympio et al. have studied the effect of a teaching intervention on the performance of residents managing checkout procedures on the anesthesia machine.8 Residents in the control group were videotaped performing an anesthesia machine check twice before their videos were reviewed with them. Residents in the test group were also videotaped twice, but they received an instructional review of their videotaped performance between assessments. The performances of both groups were evaluated using an institutional anesthesia machine checklist. The test residents (81% criteria checked) significantly outperformed the control residents (63% criteria checked). Similarly, in our study, the junior simulation residents checked 84% of the checklist items while the senior control residents checked only 54% of the items. Clearly, intensive training correlates with improved performance.

Of greater interest is our finding that simulation residents continued to outperform control residents when retested in their senior year of training (two to four years after receiving the teaching session). Initially, their performance may have been due to the proximity of their training session to their assessment session. However, simulation residents who repeated the study in their senior year continued to outperform the control residents on the checklist. The attrition in their repeated scores likely attests to the need for a review partway through resident training. The optimal interval between initial and refresher training is uncertain and cannot be determined from this study.

In keeping with the trends seen in the checklist scores, the junior simulation residents detected more preset machine faults (80% detected) than the control residents (64% detected). In most cases, machine faults were not detected because that specific step of the checkout procedure was not performed (i.e., forgotten by the resident). Unlike the decline in checklist scores, the number of machine faults detected by the simulation residents increased by 8% in their senior year when they repeated the study. We would have expected their performance to show attrition due to need for refresher training. One explanation for this increase is that the simulation residents may have remembered the preset machine faults, although this is unlikely given the prolonged interval between testing. Another explanation is that detecting machine faults is a practical skill that may be better performed than recalling the fine points of a detailed checklist. The fact that the simulation residents were tested on the machine faults three weeks after receiving the teaching session and tested again in their senior year could also have contributed to their superior performance. This viewpoint is supported by the work of Karpicke and Roediger,10 who showed that students’ retrieval of knowledge under repeated test conditions significantly improved their knowledge retention compared with knowledge retrieval that involved studying without testing. In our study, it is possible that retrieval contributed towards the simulation residents’ learning and boosted their performance in their senior year.

A potential criticism of our study is that we did not test the junior residents on their knowledge of the anesthesia machine check before they received their experiential teaching session. Thus, one could speculate that the junior residents may have possessed a superior level of knowledge of the anesthesia machine before receiving the teaching intervention. However, this was an unlikely possibility given that these residents were at the beginning of their anesthesia training.

Others11 have specified that it is essential to clinician use and acceptance to have a concise anesthesia machine checklist which can be completed easily and in a short time period. Time, however, is not an accurate assessment of the completeness of a machine check. In our study, although the simulation residents took more time to check the machine, the machine check was more thorough than that of the control residents.

We used a checklist tailored to the specific anesthesia machine used in our institution. There is no universally accepted definition of an ideal anesthesia machine checklist. The Canadian Anesthesiologists’ Society Guidelines to the Practice of Anesthesia - Pre-anesthetic Checklist consists of 46 items, 34 of which are specific to the anesthesia machine. Several of the 34 items have two steps combined into one (e.g., “Check unidirectional valves and soda lime”).12 The difference in the length of various checklists is due to the inclusion of “extra” items, for instance, our checklist includes a low pressure system check. Further support for the use of a tailored machine-specific checklist comes from the American Society of Anesthesiologists who recently condensed the 54-step 1993 FDA Anesthesia Machine Pre-Use Check into a 14-item guideline that serves as a template for the development of an institution-specific anesthesia machine checklist.9

There are several limitations to our study. Resident assessment was not blinded. We considered videotaping the residents, but we ultimately considered that a static camera angle would not capture the nuances of their check given our clinical observations of how frequently residents would circle a machine during their checkout procedure. Also, with a well-designed checklist, there should be no ambiguity as to whether a task was performed. Just over half of the residents repeated the study in their senior year, and perhaps these residents self-selected themselves into a group that was particularly knowledgeable about the anesthesia machine. It could be argued that simulation residents did better because they received additional teaching; however, during their residency training, the control residents undoubtedly would have received additional informal teaching in the operating room on the anesthesia machine.

Proponents of simulation-centre education cite the many advantages of learning in a simulated environment,13–15 including 1) repetitive and deliberate practice of skills with opportunities for learner feedback; 2) opportunity to learn on an individualized basis with individualized feedback (learner centred); 3) opportunity to learn in the clinical context (contextual learning); 4) learning that is no longer restricted to “chance” clinical encounters; and 5) opportunity to learn skills in a controlled environment without harm to patients. Simulation residents in our study likely benefited from many of the above advantages. The focus of many studies in the simulator has been on the improved acquisition of skills, and studies examining skill retention have followed learners for only one16 to 1417 months after the educational intervention. In our view, the retention of knowledge and skills shown by the simulation residents in this study (up to four years) is beyond that reported previously in the literature.

Despite studies showing poor performance on the anesthesia machine check both in clinical practice4,18 and in a simulation environment,7,19 few studies have assessed the effectiveness of teaching the anesthesia machine check to residents. Our results suggest that a brief experiential training session associated with a boost from repeated testing made even the most junior residents achieve results superior to senior colleagues after a five-year residency. Although there was some attrition in their performance, the residents retained most of their skills for up to four years as they continued to outperform compared with controls.