Background

Evidence for efficacy of an intervention may be clearly established, but its expected effectiveness outside of a clinical trial setting, the values of decision makers, and resources available at a given point in time may all affect how that intervention is brought into generalized clinical care [1]. These may vary across different countries and decision-makers and this may lead to different interpretations of the evidence [2–5]. Hence different criteria, guidelines and appropriateness ratings often need to be developed in different countries, which may then guide the clinical audit of interventions and outcomes.

The first appropriateness study on the use of carotid endarterectomy (CE), prompted by concerns of its unnecessary use, suggested that only a third of CE procedures were appropriate [6]. It was published in 1988, prior to the large randomized controlled trials [7–10] that provide our current evidence base for the majority of indications for this procedure. In spite of the proliferation of the highest level of evidence available to guide patient management, controversy still exists as to what constitutes appropriate use of the procedure, especially in those with asymptomatic disease [11, 12]. This may explain some of the marked variation in the rate of usage of CE between countries and regions [13, 14].

The objectives of this paper were: firstly, to describe the development of contemporary appropriateness ratings for carotid endarterectomy according to a panel of Canadian experts; and secondly, to develop an audit tool for chart review based on these ratings. The ratings generated from this process are likely to be of particular relevance to health systems that resemble Canada's single payer system, and the tolls that we provide will facilitate the conduct of CE appropriateness audits as a means of ensuring optimal utilization of this procedure.

Methods

Appropriateness methodology

The RAND/ UCLA Appropriateness methodology has been described in depth elsewhere [1, 2, 15–18]. In brief, a comprehensive literature review is conducted collecting all available evidence for a given diagnostic or therapeutic intervention. A set of scenarios is developed covering all possible clinical situations under which that intervention might be performed. A panel of experts is asked to independently rate each scenario guided by the evidence presented. The rating scale ranges from 1 to 9; 1 to 3 indicating an inappropriate indication, 4 to 6 an uncertain indication, and 7 to 9 an appropriate indication. (For all definitions see Appendix) These initial ratings are collated. The panel is then convened and the participants provided with their initial rating and the median rating of the group for each scenario. Under the guidance of a chair, the group discusses each scenario with the relevant evidence and individually re-rates each scenario. The median of these final ratings is used to determine the appropriateness of each scenario according to the schema as presented above. Given that none of the scenarios, provoked disagreement among the panellists according to the RAND definition this played no role in the allocation of appropriateness.

The Medline database was used as the primary route for searching for relevant material for the literature review for this study. Evidence was rated as described elsewhere [19]. Where evidence was available for a given indication from randomized controlled trials, no further literature was presented. If this level of evidence was unavailable, literature from the next available evidence hierarchy was presented.

The nine panel members were selected from nominations made by presidents of the various Canadian national societies representing stroke neurology (two members), neurosurgery (two), vascular surgery (two), internal medicine (one), family medicine (one) and neuroradiology (one). The consensus conference proceedings were taped and transcribed.

Analyses

A descriptive analysis of the final appropriateness ratings of both symptomatic and asymptomatic stenoses by degree of internal carotid artery stenosis was performed. For all scenarios in this analysis, conventional angiography was the imaging modality assumed to have determined the degree of stenosis. The transcribed discussions of the panel were used to help to identify the 'key elements of reasoning' behind the panel's decisions. These 'key elements of reasoning' are presented in narrative form in the text of the Results section to accompany the quantitative reporting on panel ratings for the clinical scenarios.

A Microsoft Access® Database was developed to enable easy collection of the necessary clinical data to determine appropriateness from patient charts. Data can be entered directly into this database. In parallel, we developed an algorithm written in SAS version 8 for transforming the raw data gathered from chart review to final appropriateness ratings. We make both the Microsoft Access® Database and the SAS code available for download from the Internet [20].

A reviewer used the database to collect the clinical data by chart review on all carotid endarterectomies performed in the Calgary Health Region in the period of 1997 – 2001, a total of 801 cases. The data collected were then run with the SAS code and its ability to classify the cases tested.

Results and Discussion

Symptomatic carotid stenosis

The final appropriateness ratings for symptomatic internal carotid artery stenoses are summarised in Table 1. The ratings for severe (70 – 99%) symptomatic stenosis are the highest of any group. The panel felt that a majority of patients will benefit from treatment, because the magnitude of benefit conferred by surgery over medical treatment [2] has been shown to be so great that the procedure is likely to be of benefit even when performed by surgeons with a higher complication rate than in the trials.

Table 1 Final appropriateness ratings for carotid endarterectomy for symptomatic stenosis by degree of stenosis.
Full size table

Appropriateness ratings ranged from appropriate to inappropriate for moderate (50 – 69%) symptomatic stenosis. This reflected the reduced margin of benefit for CE when compared to severe stenosis in the trials [3, 4], and the higher surgical complication rates seen in real-life practice [21]. Remote transient events, either hemispheric or retinal, and early operation after moderate stroke were considered to be of uncertain benefit. The presence of a contralateral occluded artery tended to lead to an indication being deemed less appropriate.

CE for those with mild (0–49%) stenoses was never rated as appropriate. Indications thought to be of uncertain benefit were multiple or crescendo hemispheric TIAs in association with an ulcerated plaque. The evidence showing that the presence of a symptomatic, irregular plaque was associated with a higher risk of recurrent stroke was a major factor that guided the panel in this decision [22]. However, there was a disparity of opinion among panellists, which did not meet the RAND definition of disagreement, as to rating of these particular indications, displaying the concerns raised by unplanned subanalyses of randomized controlled trials, which may be misleading [23].

Symptomatic acute occlusions were rated as borderline uncertain/ inappropriate, whereas operating on a chronic occlusion was always deemed inappropriate. The difference between these two indications relates to the limited evidence from case series showing the possibility of improvement in outcome from stroke with emergency endarterectomy for acute occlusion [24].

Asymptomatic carotid stenosis

Table 2 shows the final appropriateness ratings for asymptomatic internal carotid artery stenosis. For asymptomatic stenoses less than 60%, all agreed that CE was inappropriate. There was a wide range of opinion on the ratings for endarterectomy for stenosis greater than 60% and the scores changed considerably between the two rating sessions, moving from borderline appropriate to a final rating of uncertain benefit. Only the rating of severe asymptomatic stenosis with brain infarct present on brain imaging approached the definition but did not meet the RAND definition of disagreement. Otherwise only a single rater viewed these scenarios as an appropriate. This relates to concerns with the generalization of the results of the Asymptomatic Carotid Atherosclerosis Study (ACAS) [5]. The rate of stroke and/or death in the surgical group of that trial was 2.3%. Case series have been unable to match this exceptionally low complication rate, with rates ranging from 2.5% [25] to 5.6% [26]. Most of the panel felt that in real life the marginal benefit seen in the ACAS trial would not be seen in real life. However, in view of the positive randomized trial evidence, they felt that at worst this be rated as an uncertain indication. It was felt that the results of the then ongoing Asymptomatic Carotid Surgery Trial would help to resolve the place of carotid endarterectomy for asymptomatic stenosis.

Table 2 Final appropriateness ratings for carotid endarterectomy for asymptomatic stenosis by degree of stenosis.
Full size table

Carotid endarterectomy associated with coronary artery bypass grafting

Table 3 shows the final appropriateness ratings where carotid endarterectomy is performed either with or prior to coronary artery bypass grafting. The imperative to combine procedures is derived from anecdotal experience that those patients with asymptomatic carotid stenosis are at higher risk of stroke following coronary artery bypass grafting. However, case series in the literature both support and refute this [27–30]. The panel discussed a report that the combination of both procedures in a recent large case series appeared to put patients at an unacceptable risk of stroke or death (17.4%) [22]. Therefore, in all but the highest degree of asymptomatic stenosis, the panel thought it was an inappropriate indication for surgery, and even in that situation, an uncertain indication only.

Table 3 Final appropriateness ratings for carotid endarterectomy combined with coronary artery bypass grafting (CABG) for asymptomatic stenosis by degree of stenosis.
Full size table

Performance of the appropriateness rating algorithm

Of the 801 CE cases examined, 145 (18.1%) were performed for asymptomatic carotid stenosis. When these data were run with the Access database and SAS code that we developed, 541 were deemed appropriate (67.5%), 61 inappropriate (7.6%), 175 uncertain (21.9%). The most common reasons for inappropriate CE were as follows: asymptomatic stenosis less than 60% (28 cases), presence of concurrent medical co-morbidity (dementia, recent myocardial infarction) (18 cases), symptomatic stenosis <50% (five cases), and chronic occlusion (eight cases).

Only 24 cases (3%) were not classified. The reasons for the unclassified cases are as follows: no carotid artery imaging results available (thirteen cases), no information available regarding the degree of contralateral carotid artery stenosis in the presence of a moderate symptomatic stenosis (five cases), endarterectomy ipsilateral to ischaemic symptoms or side not identified (six cases).

Conclusions

The current study involved an all-Canadian panel of experts from the whole spectrum of specialties (Family Medicine, Internal Medicine, Neuroradiology, Neurology, Neurosurgery and Vascular Surgery) that are involved in the care of patients with carotid artery disease. Ratings of a balanced panel have been shown to reflect the opinion in the broader physician community [31] and to avoid the bias inherent in having only practitioners of the intervention doing the rating [12]. Each panellist was nominated by his/her respective specialty society and represented Canada geographically well with panellists from Nova Scotia to British Columbia. Their interpretation of the evidence could therefore be expected to have included a representative view of the current values and resource implications of the Canadian healthcare system.

These ratings identify ongoing areas of uncertainty that could be used to identify priority areas for research. The combination of coronary artery bypass grafting in association with carotid endarterectomy requires further close examination, especially given the high complication rates seen in the Multistate utilization study [21]. CE for asymptomatic carotid stenosis remains an area of uncertainty. Hopefully, the completed Asymptomatic Carotid Surgery Trial will shed further light on the efficacy of CE for asymptomatic disease, but it may be that further work is merited to risk stratify those for whom CE is appropriate. The ratings presented here, however, represent the Canadian viewpoint on the decision to perform CE for asymptomatic carotid disease until the Asymptomatic Carotid Surgery Trial is published. These will remain valid in determining the appropriateness of all CE performed prior to the date of its publication and for some time afterward, until clinical experts and guideline-producing bodies fully reappraise the role of CE in asymptomatic carotid disease after the new trial's publication.

This ongoing area of uncertainty with regard to CE for asymptomatic internal carotid artery stenosis led to the conservative ratings of this Canadian Panel, rating it an intervention of uncertain benefit at best. This is in keeping with international opinion leaders [12]. In contrast, this is discrepant from guidelines [32], appropriateness reviews [33–35] and practice patterns [20, 36] from the United States, where the approach toward asymptomatic disease is more aggressive with surgical intervention being seen as standard of care. The synthesis of the evidence by this panel in the form of the appropriateness ratings may therefore be of considerable value to clinicians and policy-makers in health care systems that resemble Canada's single payer system, as opposed to the more procedurally aggressive American system.

Perhaps most importantly, this paper provides access to downloadable audit tools that readers can use to rapidly implement carotid endarterectomy appropriateness audits in their own clinical setting [20]. Both the data collection template and accompanying SAS code are easy to use, and will enable interested parties to perform such CE appropriateness work to compare their local standards of care with the latest Canadian interpretation of the available evidence.

Appendix

Definitions

Appropriate – where a procedure is worth doing if the expected medical benefit to the patient (health status, quality of life, longevity) exceeds the expected negative consequences to the patient (pain, disability, risk of death).

Inappropriate – where a procedure should not be performed if the expected negative consequences to the patient exceeds the expected medical benefit to the patient.

Uncertain – where the net result of a procedure's expected medical benefit and negative consequences are equivocal.

Agreement – scenarios where no more than two of the ratings are outside of the 3-point region containing the median result.

Disagreement – scenarios where three or more readings are in the 1-to-3 region and three or more readings are in the 7-to-9 regions.

Scenario definitions

Asymptomatic carotid artery stenosis – Carotid stenosis that has been identified through imaging, which on thorough history and examination has no ascribable symptoms.

Symptomatic carotid artery stenosis – Carotid artery stenosis, which has been identified following either transient monocular blindness (retinal TIA) or retinal infarction, or transient cerebral ischemia (hemispheric TIA) or stroke.

Crescendo TIA – recurrent transient cerebral or retinal ischaemia in the distribution of the carotid artery, characterized by a definite change in pattern, such as, (1) increased frequency, multiple episodes in a single day, over several days, or a cluster of spells over the course of several days, (2) increased duration, that is, spells lasting longer than the primary event, or several hours in duration, or (3) increased severity, that is, spread in the distribution of ischaemia with greater or new motor, sensory, speech, or visual defects.

Multiple TIA – multiple TIAs that do not match the definition of crescendo TIAs.

Moderate stroke – corresponds to a disability of 3,4 or 5 on the modified Rankin Scale.

Mild stroke – corresponds to a disability of 1 or 2 on the modified Rankin Scale.