Carotid endarterectomy reduces the long-term stroke/death rate of symptomatic patients with a moderate/severe carotid artery stenosis.1 Compared with carotid artery stenting, carotid endarterectomy has a lower perioperative (0-30 days) stroke rate and therefore remains the preferred intervention for symptomatic patients with a moderate/severe carotid artery stenosis.1,2 This surgery can be performed under regional anesthesia, which allows clinical monitoring of the cerebral functions during the carotid artery cross-clamping.3 Compared with general anesthesia, regional anesthesia reduces the need for shunt insertion during carotid artery cross-clamping.4,5 A liberal use of shunting may increase the long-term incidence of restenosis;6 however, not all patients/anesthesiologists/surgeons will agree to proceed under regional anesthesia. It is therefore important to know which monitors are suitable to use when determining which patients require a shunt during carotid artery cross-clamping when surgery is performed under general anesthesia. The aim of the present meta-analysis is to compare the ability of different types of brain monitoring systems vs clinical monitoring of the brain function to detect cerebral ischemia during a carotid artery cross-clamping under regional anesthesia. More specifically, the null hypothesis is that the differences in sensitivity and specificity between the clinical neurological evaluation and stump pressure, transcranial Doppler (TCD), cerebral saturation, electroencephalography (EEG), or evoked potentials will not exceed 1% (within 95% credible intervals).

Methods

The authors developed and approved a protocol before starting this study. In May 2012, a search was conducted (J.G.) in PubMed (from January 1950), Cochrane Central Register of Controlled Trials (April 2012), Cochrane Database of Systematic Reviews (2005 to April 2012), EMBASE (1988 to 2012 Week 18), and Ovid MEDLINE® (1945 to May Week 1 2012) for prospective trials of 20 patients or more where a cerebral monitor was compared with clinical brain monitoring during a carotid endarterectomy performed under regional anesthesia. Reference lists of all retained studies were also checked. Using the Cochrane Collaboration’s tool (Table 1) as “yes”, “no”, or “unclear”, the studies were evaluated on the information contained in the reports without any assumption.7 Possible duplicate publication was ruled out by checking locations and dates of data collection. Data (true positive, true negative, false positive, and false negative) were extracted independently by the two investigators, and conflicts were resolved by agreement. The authors kept only their best results (lowest sum of false negative + false positive) when the same device was tested with more than one criterion on the same study population. A positive test under regional anesthesia could be defined as: an altered conscious state (unable to answer questions or perform a simple mental task, e.g., counting backward from 100), slurred speech, confusion, severe agitation, aphasia, loss of consciousness, seizure, or motor weakness on the side contralateral to the surgery (e.g., leg and foot paralysis, unable to squeeze a squeaky toy or a bag filled with air or saline and connected to a transducer). Two analyses were performed, one where studies with a zero cell were dropped and a second (to the reviewer’s request) where 0.5 was added to all cells of studies with zero cells only.

Table 1 The Cochrane Collaboration’s tool for assessing the quality of diagnostic studies
Full size table

Data were analyzed for risk of bias assessment with RevMan 5 Version 5.0 (The Nordic Cochrane Centre, Copenhagen) and evaluated for sensitivity, specificity, heterogeneity, positive (LR+) and negative (LR−) likelihood ratios, and receiving operator curves with Meta-DiSc® Version 1.4 (Madrid, Spain) (http://www.hrc.es/investigacion/metadisc.html). We assessed heterogeneity with the I-squared statistic, and random effects models were used for all pooled estimates (sensitivity, specificity, diagnostic odds ratio, and likelihood ratios). The JMP® software (SAS Institute Inc., Cary, NC, USA) was used to construct plots of posterior vs prior probabilities (0.001, 0.1, 0.14, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, and 0.99) from pooled LR+ and LR−. Results were then transferred to Prism (GraphPad Software Inc.; www.graphpad.com) to create Fig. 5. Classical fail-safe numbers to assess publication bias were calculated with Comprehensive Meta Analysis Version 2.2.044 (www.Meta-Analysis.com).

Results

A flow chart for the selection of the studies is shown in Fig. 1, and an evaluation of the quality of the studies is shown in Fig. 2. A list of all articles retrieved is available as Electronic Supplementary Material (Appendix). Details of the study are presented in Table 2. Data were extracted for 4,664 measurements taken from 29 studies (transcranial Doppler = 739; cerebral saturation = 320; stump pressure = 2,549; EEG = 742; evoked potentials = 187; jugular venous saturation = 79; and jugular venous lactate = 48) (Fig. 3).8-36 Agreement before discussion was achieved in 92 of 156 extractions (true positive, false positive, true negative, and false negative for each test) for a kappa statistic of 0.59. All discrepancies were resolved after discussion. Sensitivity, specificity, area under the curve, diagnostic odds ratio (DOR), LR+ and LR− are given in Table 3. Inclusion of studies with zero cells improved most DORs (except EEG) but did not change the fact that the two best monitors were TCD and EEG (Table 4). The classical fail-safe numbers (i.e., number of negative studies required to render the DOR non-statistically significant) were 99 for TCD, 153 for EEG, and 15 for cerebral saturation. A moderate amount (I-square = 56.2%) of heterogeneity was found on the DOR for EEG. A meta-regression revealed that the DOR improved significantly with the number of channels used (P value = 0.03) (Fig. 4). Posterior probabilities for sequential testing with stump pressure 25 mmHg and each of TCD, cerebral saturation, EEG, or evoked potentials are given in Fig. 5. Assuming a 14% incidence of shunt requirement,5 the chances of requiring a shunt when two tests are positive would be 90.0% for stump pressure 25 mmHg and TCD; 77.5% for stump pressure 25 mmHg and cerebral saturation; 93% for stump pressure 25 mmHg and EEG; and 75.0% for stump pressure 25 mmHg and evoked potentials. The chances of requiring a shunt when two tests are negative would be 1.4% for stump pressure 25 mmHg and TCD; 2.5% for stump pressure 25 mmHg and cerebral saturation; 2.4% for stump pressure 25 mmHg and EEG; and 1.5% for stump pressure 25 mmHg and evoked potentials.

Fig. 1
figure 1

Flow chart for the selection of the studies. EEG = electroencephalography

Full size image
Fig. 2
figure 2

Evaluation of the quality of the studies included in the analysis according to the Cochrane Collaboration’s tool for systematic reviews of diagnostic tests. Green = % of the studies for which this specific item was judged to be adequate; red = % of the studies for which this specific item was judged to be inadequate; yellow = % of the studies for which there was not enough information in the article to judge this item

Full size image
Table 2 Characteristic of included studies
Full size table
Fig. 3
figure 3

Forest plots of diagnostic odds ratios for transcranial Doppler (TCD), electroencephalography (EEG), cerebral saturation (CS), and evoked potentials (EP). The diagnostic odds ratio (ratio of the odds of positivity in disease relative to the odds of positivity in the non-diseased) expresses the strength of the association between test result and disease. Inconsistency (as measured by the I-square value) is the percentage of variation in effect from one study to another that cannot be explained by chance alone: 25% is low, 50% is moderate, and 75% is high. I-square was 24.9% for TCD, 56.8% for EEG, 30.8% for CS, and 69.1% for EP

Full size image
Table 3 Results for all monitors
Full size table
Table 4 Results for all monitors adding 0.5 to all cells of studies with zero cells
Full size table
Fig. 4
figure 4

A meta-regression revealed that the diagnostic odds ratio for EEG increases with the number of channels used (P = 0.03)

Full size image
Fig. 5
figure 5

Graph of the posterior probabilities vs the prior probabilities for sequential testing with stump pressure with a cut-off point at 25 mmHg, followed by transcranial Doppler (TCD), cerebral saturation (CS), electroencephalography (EEG), or evoked potentials (EP) for a positive or negative (dashed lines) test. Points are the results for a prior probability of 14%

Full size image

Discussion

Our results show that a combination of stump pressure and either TCD or EEG appears to deliver the best results for detecting brain ischemia during carotid artery cross-clamping. To achieve best results, EEG should be used with a high number of channels. Ambiguity of blinding was an obvious flaw for most of the studies included. The effect of absence of blinding on the results is difficult to evaluate precisely, but it is certainly possible that the capabilities of the various monitors studied to detect cerebral ischemia were overestimated. Bias in systematic reviews of diagnostic tests would increase the sensitivity (no systematic effect on specificity).7,37

Studies with empty cells are problematic. The addition of a fraction to those cells or to all cells has been proposed as a solution to circumvent this issue, but this technique is controversial as it may increase heterogeneity (Table 4) and introduce bias in the results. In developing the protocol, we therefore specified that all studies with an empty cell would be dropped in the final analysis; however, by doing so, we may have underestimated the accuracy of the monitors. The exact cut-off point used to determine the shunt requirement varied from one study to another. Instead of evaluating a cut-off point determined a priori, the investigators often gave us a cut-off point that best fit their data. Changing the cut-off point will affect the monitor’s performance. For example, a stump pressure of 25 mmHg may be more specific but less sensitive than a stump pressure of 40 mmHg. Based on our protocol, we chose to extract data from each study that gave the best results (lowest sum of false positive + false negative) so as to keep the best accuracy of each monitor. This may have overestimated the performance of the monitors evaluated. The definition used to define the “gold standard” (Table 2) varies from one study to another. This may have affected the performance of the “gold standard”; however, regardless of the sign or symptom, we consider any new neurological sign/symptom (global or focal) that appears during carotid artery cross-clamping (provided that the patient is not oversedated) to be interpreted as suboptimal brain oxygenation until proven otherwise. Consequently, our a priori definition was extensive, and no study was excluded for this specific reason. Finally, the search was limited to the English language. The effect of language restriction has been evaluated and, for conventional intervention (as opposed to complementary and alternative medicine), a search limited to the English language is unlikely to introduce a significant bias into the conclusions.38

Taken separately, none of the monitors can detect cerebral ischemia to the same extent as regional anesthesia. Association of stump pressure with TCD or EEG seems to be the best combination if, fortuitously, results of both tests are on the same side (positive vs negative) (Fig. 5). The inherent limits/costs of tests may also affect choice. For example, approximately 10% of patients lack a temporal window that would allow TCD monitoring. Alternate approaches have been proposed to circumvent this problem.17 The close proximity of the monitor to the surgical site has also been a problem, with some investigators reporting probe displacement during the surgery. Simplified EEG (BIS monitor) proved unfit for this task, and because the interpretation of a conventional EEG requires specialized knowledge, the availability of the technician/neurologist limits the use of an EEG for this purpose. When EEG is chosen, our meta-analysis suggests that a high number of channels should give the best accuracy (Fig. 4). Some drugs used to induce and maintain general anesthesia will affect EEG and evoked potentials, making the ability of these two monitors to detect brain ischemia in patients under general anesthesia uncertain.

The ultimate decision whether or not to insert a shunt will also involve consideration of other variables, such as specific difficulties of shunt insertion (e.g., long plaques extending distally into the cranium, small vessels etc.), stenosis/occlusion of the collateral vessels, capability of the patient to support an increase in systemic arterial blood pressure (failing heart), or timing of ischemia (a brief period of ischemia close to restoration of blood flow may be less risky than a shunt insertion).

Finally, it is important to bear in mind that accuracy of tests constitutes a surrogate marker for important outcomes. In our study, the evaluation was based on cerebral ischemia as attested by a variety of clinical signs and symptoms, while the obvious important outcome would be a change in the stroke/death rate (critical to a decision or 7-9 on a 9-point rating scale).39 Assuming that strokes could be prevented by a better diagnosis of cerebral ischemia during artery cross-clamping implies not only that no (or a low rate of) bad consequences would be expected from the shunt placement itself but also that ischemic strokes constitute a sufficiently important fraction of all strokes to make a clinically relevant difference or that an early diagnosis of embolic strokes would change the overall prognosis. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group considers that “accuracy studies typically provide low-quality evidence for making recommendations owing to indirectness of the outcomes, similar to surrogate outcomes for treatments.”39 The second important outcome here (which justified undertaking the present study) was that a more accurate diagnosis of cerebral ischemia could decrease unnecessary shunt placements in patients operated under general anesthesia, a practice which itself may lead to restenosis (based on a large retrospective study) (important to a decision or 4-6 on a 9-point rating scale). Nevertheless, a low (TCD) or moderate (EEG) amount of heterogeneity on the DOR coupled with uncertainty around the results (wide confidence intervals) reduces the quality of the evidence. Therefore, a prospective randomized study evaluating the efficacy of our proposed strategy to reduce the rate of long-term restenosis or costs associated with shunt placements (without increasing the stroke/death rate) would be required before making any firm recommendation.

In conclusion, no monitor can reproduce the detection of brain ischemia achievable with regional anesthesia. A combination of stump pressure and either TCD or EEG appears to give the best results. For EEG, a high number of channels should be used. A randomized clinical trial evaluating this strategy is required before any firm conclusion can be drawn.