Since the introduction of curare in anesthesia practice,1 paralysis with neuromuscular blocking drugs (NMBDs) has been an important component of general anesthesia. Successful recovery from neuromuscular blockade at the end of a surgical procedure is essential to avoid residual paralysis and its deleterious consequences.2-5 Pharmacologic agents are often used to accelerate the recovery from neuromuscular blockade. The timing and dose of reversal agents are guided by the degree of spontaneous recovery from NMBDs.6,7 The most common monitoring modality conventionally used for intraoperative management of NMBDs is the train-of-four (TOF) count.8-10 While the quantitative assessment of the TOF ratio (i.e., ratio of fourth to first response) has been shown to be the most accurate, a simple TOF count is often used clinically as a rough estimate prior to reversal to guide the timing and dose of reversal agents. Indeed, when reversal is properly timed and dosed, all patients are expected to achieve four twitches without fade. Therefore, as a rule, assessment of the TOF count is meaningful primarily before the administration of reversal agents. In recent studies on the reversal of NMBDs from rocuronium using sugammadex, the TOF count was objectively measured by the TOF-Watch® SX to determine the reversal dose.11-14 In clinical practice, most anesthesia providers use a peripheral nerve stimulator (PNS) for intraoperative assessment of the TOF count. If the results of qualitative monitoring with a PNS are used to determine the timing and dose of reversal agent, it is important to know if there is concordance between qualitative monitoring of the TOF count and the TOF-Watch SX assessment, since the dosing recommendations were developed using the TOF count measured with the TOF-Watch SX. If the two modalities are not concordant, such practice may lead to inappropriate dosing of drugs like sugammadex, resulting in either residual paralysis in the postoperative period or unnecessary costs. On the other hand, neostigmine dosing guidelines are based on the TOF count measured with a qualitative PNS.15 Use of the TOF-Watch SX to measure the TOF count could also result in inaccuracies if there is no concordance between the two methods.

The primary goal of this study was to estimate and describe the percent agreement and the percent over and under estimation of the TOF count by provider compared with the TOF-Watch SX measurements. Secondary aims were to test the null hypothesis that there was no bias (i.e., positive and negative differences were equally likely) and the null hypothesis that bias did not depend on the type of provider. We did not have an a priori directional hypothesis for either premise. Finally, we wanted to estimate the time between the detection of two twitches by the TOF-Watch SX vs the anesthesia provider, since the reappearance of the second twitch is commonly used as a threshold for the use of sugammadex to reverse rocuronium-induced neuromuscular block.

Methods

The University of Washington Institutional Review Board (Seattle, Washington) approved the study in February 2013. Written informed consent was obtained from all subjects, and patient and provider identifying information was not retained. This prospective observational cohort study was conducted at Harborview Medical Center, a level I trauma centre and referral hospital for five states in the Pacific Northwest. From June 4, 2013 to July 31, 2013, we enrolled 90 patients with American Society of Anesthesiologists (ASA) physical status I-III who were free from an underlying neuromuscular disorder or neuropathy and scheduled to undergo elective surgery with anticipated use of nondepolarizing muscle relaxants (rocuronium or vecuronium). Exclusion criteria included patients who were younger than 18 yr or older than 80 yr of age, patients who were pregnant, patients where intraoperative access to an upper extremity was unattainable for monitoring the TOF count, and non-English speaking patients. Certified registered nurse anesthetists (CRNAs), residents with attending physician supervision, or attending anesthesiologists provided anesthesia care. Patients received standard anesthesia care and monitoring, including electrocardiography, pulse oximetry, end-tidal carbon dioxide concentration, and noninvasive blood pressure monitoring. All patients had forced-air warming in the intraoperative period. The relevant anesthesia providers were informed that their patients were enrolled in a study, that usual care should be provided, and that their role was limited to providing TOF counts at the request of study staff.

Study procedures

Following administration of the muscle relaxant, two standard electrocardiogram electrodes were placed approximately 4 cm apart along the ulnar nerve over the skin of the distal forearm. A TOF-Watch SX (Bluestar Enterprises, Omaha, NE, USA) was used on all subjects. The acceleration transducer was positioned on the distal phalanx of the thumb via a TOF-Watch hand adapter (Organon Ltd, Dublin, Ireland) that applied a constant preload to the thumb and allowed a reproducible baseline thumb position. We used 50-mA TOF stimulation without calibration to deliver four pulses every 15 sec; each pulse had a frequency of 2 Hz and a duration of 0.2 msec. The evoked response of the adductor pollicis was evaluated. The said anesthesia care providers were familiar with neuromuscular monitoring and used it frequently. Neuromuscular monitoring is used routinely at our institution for all patients who receive NMBDs, and therefore, our providers gain substantial experience. The anesthesia providers were not informed about the number of measurements to be made or other details of the protocol and were blinded to the TOF count displayed on the TOF-Watch SX. They were told that several TOF count measurements would be requested during the case and were instructed to use their usual clinical practice of visual, tactile, or combined (visual and tactile) assessment to state the observed TOF count to the study team member. All measurements were based on thumb twitches.

The researcher watched the TOF-Watch SX, and as soon as it changed to the next higher TOF count value, the researcher asked the provider to give a subjective assessment of the TOF count. When the provider was ready to assess the TOF count, we recorded the TOF-Watch SX measurement, and the provider evaluated the TOF count 15 sec later when the next stimulation occurred. The researcher then asked the provider to reassess the TOF count two to five minutes later. Thus, the anesthesia provider was asked to assess the TOF count twice at each level of TOF-Watch SX count such that there was a maximum of ten data points collected for every patient (two provider counts at machine count 0 and at each incremental machine count). We also recorded the time when the anesthesia providers gave the TOF counts.

The ten measures were recorded during the offset of the first dose of neuromuscular relaxant starting at a TOF-Watch SX count of 0 approximately ten minutes after tracheal intubation. The study team aimed to have a minimum gap of two to five minutes between two contiguous assessments, whenever possible. If the anesthesia provider was busy with other tasks related to patient care or documentation of care, an assessment was not required. If the anesthesia provider decided to administer more neuromuscular relaxant prior to collecting all ten data points, data collection was terminated at that time and concluded for the case.

Data collection

The demographic variables collected included age, sex, height, weight, body mass index, surgical procedure, and ASA physical status. The type of muscle relaxant, dose, and time of administration were recorded, and we also noted the type of anesthesia provider (resident, CRNA, attending anesthesiologist). For each of the ten observations, we recorded time, TOF count (by TOF-Watch SX and by provider), and whether the provider used visual, tactile, or a combined method to assess the TOF count.

Sample size

The primary goal of this study was to estimate the degree of concordance between the subjective counts of the anesthesia providers and the TOF-Watch SX measurements and to describe any discrepancy. With a sample size of n = 60, the standard error of the estimates of agreement would be less than six percentage points, meaning that the precision of those estimates is better than plus/minus 12 percentage points. A study sample of 60 subjects would yield 85% power for our secondary endpoints to detect a mean bias between the two measures of 0.15 and 80% power to detect a difference in mean bias of 0.3 between CRNA and resident. To account for the possibility of technical problems, subjects with incomplete data, or withdrawal, we inflated the sample size to include 90 patients.

Statistical analysis

Stata® version 13 (StataCorp, College Station, TX, USA) and IBM SPSS version 19.0 (IBM Corporation, Armonk, NY, USA) statistical software were used for all analyses. All tests were two-sided. Patient demographics and measurement times are reported descriptively. The concordance between the subjective TOF count and the TOF-Watch SX measurement at each of the ten time points is displayed in tables and graphically. In the primary analysis, we compared the two provider assessments at each level of TOF count vs the TOF-Watch SX measurement. In order to test whether bias (the difference between provider TOF count and TOF-Watch SX count) differed by type of provider, we first collapsed data from the ten measurement times into a single number for each case as follows: The mean bias for a case was computed as the mean, across the ten measurement times, of provider TOF count minus TOF-Watch SX count. These analyses included only the 63 patients for whom all ten measurements were available. Analysis of variance was used to test whether bias differed by provider type–coded as attending, resident, CRNA. Because providers may differ from each other in degree of bias, the model includes a random provider effect with providers nested within provider type.

We also estimated the number of minutes from when the anesthesia provider first rated the TOF count as 2 to when TOF-Watch SX first measured the TOF count as 2. Nevertheless, this estimation is not straightforward. The time of the first occurrence of provider TOF count = 2 was not known precisely; rather, we know a lower bound (i.e., the last time provider count = 1 was observed) and an upper bound (i.e., the first time provider count = 2 was observed) for this time; this is referred to as interval censoring. We used the “survfit” function in the R statistical analysis program to accommodate this interval censoring when estimating the median time from first provider TOF count = 2 to first TOF-Watch SX count = 2.

Results

We enrolled 90 subjects during the study period. The study flow chart is shown in the Figure. Fifteen patients were excluded from the study for the following reasons: adequate relaxation with a TOF count of 4 was not achieved within five minutes of intubation (n = 7), intraoperative positioning did not allow access to the adductor pollicis for twitch monitoring (n = 4), NMBDs were not used by the anesthesia provider after the patient consented to take part in the study (n = 3), and re-dosing of NMBDs at a TOF count of zero (n = 1). Twelve patients had incomplete data for the ten study measurements. The reasons for missing observations were re-dosing of NMBDs before return of a TOF count of 4 (n = 4), the anesthesia provider was unable to perform the TOF count assessment (n = 4), NMBDs reversal before the TOF count returned to 4 (n = 2), and technical difficulties with the TOF-Watch SX (n = 2). Observations at all ten study time points were obtained in the remaining 63 patients.

Figure
figure 1

Study flow chart

The cohort characteristics are shown in Table 1. Anesthesia care for the 75 patients included in our analysis was provided by 38 anesthesia providers, six attending anesthesiologists, 17 residents, and 15 CRNAs. Some of the residents and CRNAs provided anesthesia care to more than one enrolled patient.

Table 1 Cohort characteristics

Provider TOF counts were tactile, visual, and a mix of tactile and visual in 61%, 12%, and 27% of cases, respectively. Overall, 75% of the ratings were tactile and 25% were visual. The mean rating did not differ by rating method at any of the ten rating times (P > 0.10). Consequently, all ratings were combined for analysis regardless of rating method.

The providers made 687 TOF count assessments (including both assessments at each TOF-Watch SX count), and their distribution is summarized in Table 2. In 386 (56%) of these observations, there was agreement between the TOF-Watch SX and the subjective provider assessment, with the utmost agreement at the extreme levels of TOF count, i.e., TOF counts 0 and 4 (87% agreement). There were 409 observations at TOF counts 1, 2, and 3, with 36% agreement at these count levels. Among the 264 observations with disagreement at these levels (TOF counts 1, 2 and 3), providers assessed a higher TOF count than TOF-Watch SX in 254 (96%) observations and a lower count than TOF-Watch SX in ten (4%) observations.

Table 2 Frequency of all TOF counts by anesthesia providers and by TOF-Watch SX

The agreement between the TOF-Watch SX and the anesthesia providers for the first and second observation separately at each TOF count are presented in Tables 3 and 4 (available as Electronic Supplementary Material). They show somewhat better agreement at the first observation than at the second.

The analysis of variance showed an overall bias with provider TOF count being higher than the TOF-Watch SX count [mean (SD) bias = 0.50 (0.34); P < 0.01]. Furthermore, there were different degrees of bias among individual providers (variance component = 0.075; F = 2.61; df = 30,30; P < 0.01) but no evidence that bias differed by provider type (F = 0.16; df = 2,40.2; P = 0.85).

Analysis of the time to TOF-Watch SX counts resulted in an estimated median of 35 min from NMBDs administration to a TOF-Watch SX count of 2. The TOF count of 2 occurred earlier when measured by the provider vs the TOF-Watch SX. The estimated median time from provider TOF count of 2 to TOF-Watch SX count of 2 was 4.5 min, with an interval of ≥ ten minutes in about 15% of cases.

Discussion

In this study, we observed that it was quite common for anesthesia providers using visual and/or tactile methods to assess a higher TOF count than that measured by the TOF-Watch SX, especially for intermediate levels of spontaneous recovery from the neuromuscular blockade (TOF counts of 1, 2, and 3). It was also rare for provider count to be lower than the TOF-Watch SX count. There was reasonable agreement between the TOF count assessed by anesthesia providers and the TOF count measured by the TOF-Watch SX at the extremes of the TOF count (TOF counts of 0 and 4).

Though it is well known that anesthesia providers are unable to discern the fade as accurately as the TOF-Watch SX, studies validating the TOF count as measured by a TOF-Watch monitor are limited. Howardy-Hansen et al. reported a discrepancy between anesthetists vs transducer in the evaluation of post-tetanic count. 16 A study comparing a tactile TOF count by intensive care unit nurses with no previous experience with TOF monitoring vs a TOF count measured by a TOF-Watch monitor found that 46% of the nurses’ assessments were inaccurate.17 The nurses overestimated the TOF count in 25% of cases and underestimated the TOF count in 21% of cases. The level of concordance was comparable with our results at TOF counts of 1-3, but subjective overestimation was much more common in our study. In our institution, use of qualitative nerve stimulators are routine for all patients who receive muscle relaxants, which may not be representative for all hospitals.10 Moreover, we observed a higher degree of discordance in the second observation; which can be explained by the fact that providers tend to estimate higher counts, and therefore, we would expect a higher bias when more time is allowed for recovery.

It has been reported that subjective assessment of a twitch is possible when it is at least 5% of control height18; however, the TOF-Watch SX will not begin to display the responses if the movement is below the threshold of 3% of control twitch height.

Our study results imply that subjective detection of twitch occurs at a lower twitch height than the TOF-Watch SX monitoring thresholds. This finding is not consistent with previously reported thresholds.18,19 Further studies to redefine the twitch detection thresholds for each of the two methods may help explain our findings. Other possible explanations include: 1) Provider bias – the providers expected that the TOF count would increase over time as the NMBD effect wore off. It is possible that they reported a progressively higher count over time. 2) The data on the relative accuracy of providers compared with the accelerometer (TOF-Watch SX) in detecting a twitch is limited. It is possible that these data may need revalidation. 3) Use of the hand adapter may have resulted in placement of the acceleration transducer in a slightly proximal location compared with the tip of the thumb and may have contributed to the TOF-Watch SX detecting fewer twitches than the providers who watched/touched the tip of the thumb. 4) Anesthesia providers may have perceived responses of hand muscles other than the adductor pollicis as a thumb twitch, whereas the TOF-Watch SX measures only the adductor pollicis contraction.

Our findings have implications for the management of intraoperative neuromuscular blockade and the prevention of residual paralysis in the recovery room. Re-dosing of muscle relaxants as well as the timing and dose of reversal agents are often determined by monitoring the TOF count. Kirkegaard et al. reported that 20 min after the administration of neostigmine 70 µg·kg−1 at a TOF count of 4, the incidence of residual paralysis was 27%.7 This incidence increased to 56% when neostigmine was given at a TOF count of 2 instead of 4 without changing any of the other parameters.

In 15% of cases, the time from provider count of 2 to TOF-Watch SX count of ≥ 2 was ≥ ten minutes, which is a significant period of time in the context of residual paralysis after the use of intermediate duration muscle relaxants. In a study by Murphy et al.,20 the incidence of residual paralysis was 88% at the time of extubation, and it decreased to 32% by the time of arrival at the postanesthesia care unit. The difference between the two measurements was approximately 11 min.

Viby-Mogensen et al.21 found that anesthesia providers were unable to quantify the TOF ratio with a PNS irrespective of their experience. Consistent with this finding, we observed that the type of provider had no influence on the determination of TOF count, and residents, CRNAs, and attending anesthesiologists were equally likely to overestimate the TOF count compared with the TOF-Watch SX count.

As for the newer reversal agent, sugammadex, several studies have shown a dose-response relationship between the depth of the neuromuscular blockade and the dose needed to successfully reverse the blockade.11-14 Many of the studies used a TOF-Watch SX to determine the TOF count and used the reappearance of two twitches as the index time for the administration of sugammadex. In countries where sugammadex is available, the manufacturer recommends a dose of 2 mg·kg−1 to reverse moderate depth of block defined by spontaneous recovery of two twitches following rocuronium- or vecuronium-induced blockade.22 We do not know which is more accurate, the provider TOF count or the TOF-Watch SX count, but in our view, it is immaterial which method is “more correct”. An overestimation of the TOF count could lead to underdosing sugammadex with the possibility of inadequate reversal of neuromuscular blockade or even recurarization.23 The dosing guidelines for sugammadex may need to be revised based on how the TOF count is determined in practice. Our data emphasize the importance of an accurate assessment when the TOF-count is equal to 2.

Limitations

Our study has limitations that should be acknowledged. We used the TOF-Watch SX without calibration and with a standard current of 50 mA. We chose the default sensitivity and used the hand adapter to hold the acceleration transducer. In our view, these are the most common settings that providers would select when using the TOF-Watch SX.

It appears intuitive that the added weight of the hand adapter may limit the detection of twitch when using the visual and tactile methods, though our findings are contrary to this reasoning. Also, we followed the standard practice of placing the stimulating electrodes at each patient’s wrist. Nepveu et al. and Capron et al.24,25 have shown that stimulating directly over the patient’s hand vs their wrist leads to less direct muscle stimulation and a more accurate TOF count. We used stimulation at the wrist as it is the standard practice at our institution.

Though we monitored the recovery from NMBDs on a continuous basis using the TOF-Watch SX, we asked the provider to give us their assessment of the TOF count at certain times, namely, as soon as the TOF-Watch SX changed to the next higher value for TOF count and then again two to five minutes later. A limitation of this design is that it did not allow us to determine the earliest time point at which the manual TOF count increased. An alternative study design with frequent manual TOF count measurements could address this limitation but would likely be too intrusive to conduct in a clinical setting. Spot-checking (rather than continuous monitoring) the depth of neuromuscular block is a standard practice at our institution. The second provider assessment of the TOF count (two to five minutes after the first measurement) was aimed at reducing the chances that the TOF count may have recovered in the 15 sec between the observation by the TOF-Watch SX and that of the provider, respectively.

Although the anesthesia providers were not informed about the details of our protocol, and specifically not told about the number of measurements to be made, it cannot be ruled out that some providers surmised that there were two measurements at each level of TOF count determined by the TOF-Watch SX. Nevertheless, this most likely would have biased the providers towards better agreement.

Finally, we asked the providers to use their usual clinical approach for a subjective estimation of the TOF count (visual, tactile, or both). Brull and Silverman26 have shown that the differences between the visual and tactile means of assessment of fade are relatively small when using TOF and double-burst stimulation patterns for neuromuscular monitoring. Both subjective techniques are often imprecise in documenting complete recovery of neuromuscular function.

In conclusion, at the intermediate depths of neuromuscular blockade (TOF counts of 1 to 3), the TOF counts measured by acceleromyography using the TOF-Watch SX were not highly concordant with the visual and/or tactile assessment of the TOF count by the anesthesia providers. We also noted that the anesthesia providers often assess a higher TOF count compared with the TOF-Watch SX. There was greater concordance between the TOF-Watch SX and the provider assessment when the TOF count was 0 or 4. When deciding on the timing and dose of reversal agents to antagonize neuromuscular blockade, it is important to remember that the dosing guidelines are based on the TOF count measured by the TOF-Watch SX and to acknowledge that the subjectively estimated TOF count may overestimate that of the TOF-Watch SX.