Mark E. Linskey

Steven N. Kalkanis

Evidence based medicine (EBM) clinical practice guidelines are rapidly evolving into very important tools for improving patient care quality, improving patient clinical outcomes, smoothing out unexplained variation in care between providers, institutions, and geographic regions, as well as reducing cost of health care delivery. This special issue of the Journal of Neuro-Oncology presents the results of the first effort by the American Association of Neurological Surgeons (AANS), the Congress of Neurological Surgeons (CNS) and the Joint Tumor Section (AANS/CNS) to contract with an Agency for Healthcare Research and Quality (AHRQ)—funded Evidence-based Practice Center (EPC) to produce multidisciplinary evidence-linked clinical practice guidelines for the treatment of a condition that neurosurgery shares in common with many other medical specialties. This collaboration resulted in the methodologically highest quality EBM guidelines for the treatment of patients with metastatic brain yet produced, completed within 12 months of initiation, and published within 16 months of initiation.

Why guidelines?

Given the seemingly exponential increase in the annual volume of peer review literature, coupled with the significant time pressure associated with clinical practice, it is becoming increasingly unlikely that any one practitioner can keep current regarding all published evidence related to their specialty, no matter how much they sub-specialize. EBM clinical practice guidelines can help by providing a starting point where a systematic evidence review up to one point in time leads to practice recommendations supported by best-evidence-to-date formulated by a multidisciplinary panel of relevant specialty experts.

Clinical practice guidelines are systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific individual circumstances [1]. Advantages of following guidelines rather than expert opinion or individual randomized clinical trial (RCT) results include: that professional expertise is taken into account in aggregate, in a more systematic manner, that more “experts” are involved (diluting outlier opinions), that expert opinions are of the collected evidence rather than their own personal experience, and that the questions addressed in guidelines are more likely to be relevant and “generalizable” to routine practice situations than most inclusion/exclusion criteria of RCT’s.

Organized neurosurgery (AANS and CNS) has been involved in EBM clinical practice guideline since 1993. We have the longest experience with the severe head injury guidelines produced originally produced in 1996 in a collaboration between the AANS and the Brain Trauma Foundation (BTF) [2], with later revisions also involving the CNS. Over the last 13 years these EBM clinical practice guidelines have been adopted by the American College of Surgeons in their trauma center certification program as well as by the World Health Organization. They have now clearly been shown to improve patient clinical outcomes as well as reduce healthcare delivery costs [3, 4]. Acceptance and implementation of EBM clinical guidelines has the potential to smooth out unexplained variation in care between providers, institutions, and geographic regions without unjustified intrusion on practitioner autonomy or clinical judgment. They may also become one of the best sources for deriving disease- or procedure-specific process quality measures with a better evidence base that consensus process measures.

Guidelines differ in quality and importance

Guidelines construction involves two steps. First, a systematic means of identifying evidence and ranking the relative strengths, or quality of each study as evidence. Second, achieving panel agreement on strength of recommendation linked to the analysis of the strength of evidence for each intervention in question. Both steps are critically important and have their own drawbacks and limitations. The ultimate validity and usefulness of any guideline is critically related to three key factors: (1) the composition of the guideline panel and its process, (2) the identification and synthesis of the evidence, and (3) the method of guideline construction applied. Panel composition is crucial, both for ultimate acceptance of the guidelines by practicing physicians, and for its critical influence on the recommendation step of guideline construction [5]. Panelist’s recommendations can differ even when analyzing the same data [6–9].

In general, there are three different levels quality for guidelines construction methodology; informal consensus guidelines, formal consensus guidelines, and evidence-linked construction [10]. The first is most commonly found in guidelines produced by patient support or advocacy groups. They are often treatment algorithms produced without a systematic evidence review and grading, and the recommendations are usually not graded by strength. An example for metastatic brain tumors is the International RadioSurgery Association (IRSA), Stereotactic radiosurgery for patients with metastatic brain tumors guideline [11]. The second can be evidence-based, in the sense that a literature review is performed, but the evidence may, or may not be graded in evidence tables, and the strength of resultant recommendation are not limited by the strength of supporting evidence. The evidence tables utilized in construction are often not presented to allow for independent verification of evidence and recommendation strength linkage. Examples of metastatic brain tumor guidelines produced with this methodology include the four American College of Radiology ACR Appropriateness Criteria® guidelines for metastatic brain tumors [12], and the two National Cancer Institute (NCI), National Comprehensive Cancer Network (NCCN) Clinical Practice Guidelines in Oncology™ for metastatic brain tumors [13].

Evidence-linked methodology is the highest quality construction methodology possible. They represent the only guidelines with independent evidentiary status for EBM decision making [1]. With this methodology, the evidence tables are included for independent verification of adequacy of literature review and correctness of evidence grading. Recommendations are graded by strength, but strength of recommendation cannot exceed strength of evidence supporting that recommendation regardless of expert opinion and this can also be independently verified by review of the published document. Recommendations not exceeding strength of evidence, is crucial for limiting bias, as well as preserving clinician autonomy and flexibility of clinical judgment where restriction would be unjustified. Preventing unjustified restriction of clinical practice as well as a representative and expert composition of the multidisciplinary writing groups are the two most important features for maximizing the chance of guideline acceptance and utilization by clinicians. The evidence-linked multidisciplinary guideline published in this special issue is a historic milestone, as it represents the first effort using this formal methodology for managing metastatic brain tumors produced by a representative multidisciplinary writing panel of experts.

Practical problems

Over the last 16 years, the AANS and CNS and their associated joint subspecialty sections have participated in, completed, and approved eight different EBM clinical practice guidelines. A feasibility study by the CNS completed in 2005 revealed that each effort took on average 3 years to complete (range 2–5 years) relying solely on volunteer efforts of the physician writing group members. Given that the average EBM clinical practice guideline validity shelf life is only 5–7 years before the evidence review it is too out of date to be relied upon, it was clear that the time to completion really cuts into the useful shelf life of the resultant guideline. Each effort also involved production costs ranging between $20 and $100,000 independent of potential publication costs if the journal required publication as a paid supplement.

It was clear that a more professional solution was needed, particularly to expedite the first phase of guidelines construction involving the systematic literature review, evidence sifting and screening, formation of final evidence tables and initial grading of evidence strength based on study methodology. Rather than hire their own epidemiologists, research librarians and assistant infrastructure, as well as develop the in-house hardware and software needed for this portion, the AANS and CNS decided to try a pilot project contracting with an established EPC.

AHRQ EPCs and the process for this project

The Agency for Health Care Policy and Research (AHCPR) was established as a Public Health Service agency within the Department of Health and Human Services (HHS) in December 1989 under Public Law 101-239 [14]. It was tasked with promoting quality of healthcare, reducing its cost, improving patient safety, decreasing medical errors, and broadening access to essential services by supporting outcomes studies, and implementing their findings through the dissemination of clinical guidelines. In 1999 the name of the agency was changed to the Agency for Healthcare Research and Quality (AHRQ) [15].

The AHRQ is responsible, among other things, for compilation and publication of the national healthcare report card and helping to maintain the national guidelines clearinghouse (www.ngc.gov). However, approximately 75% of its annual budget goes towards providing 5-year grants to fund EPCs in North America, of which there are now 14 (Table 1). The EPCs are charged with reviewing all relevant scientific literature on clinical, behavioral, and organization and financing topics to produce evidence reports and technology assessments. These reports are used for informing and developing coverage decisions, quality measures, educational materials and tools, guidelines and research agendas. They also are tasked with doing research on methodology of systematic reviews [16].

Table 1 List of current AHRQ-funded evidence-based practice centers
Full size table

The AANS and CNS are not the first medical professional society organizations to consider contracting with an EPC. The first neurosurgery affiliated organization to do so was the BTF, which has contracted with the Oregon EPC at the Oregon Health Sciences University to work with them on regularly helping with revisions and updates of the severe head injury guidelines. Another non-neurosurgical example is the National Kidney Foundation which has developed a retainer contract with Tufts—New England Medical Center EPC to assist with their Outcomes Quality Initiatives guidelines effort.

The AANS and CNS sent a request for proposal to 13 AHRQ-EPCs in October 2006. Four EPCs initially expressed interest and three ultimately submitted proposals for review. The McMaster University EPC won the contract after careful study by the AANS/CNS Joint Guidelines Committee through the Spring of 2007. The contract was negotiated and finalized in the Fall of 2007. The project began in February 2008 and was completed in May 2009. The AANS/CNS Joint Guidelines Committee review and response to review were completed by August 2009, and AANS and CNS approval conferred later that month.

Advantages of EPCs

There are distinct advantages for professional societies working with established EPCs to create EBM clinical practice guidelines. Many of the AHRQ-funded EPC’s have extensive experience in all areas of EBM and many are actively involved in research efforts to advance EBM to the next level. They are professionals. This is what they do.

In our case, McMaster University is known world over for its seminal contributions to the EBM literature. Alumni and/or current faculty at McMasters include such EBM luminaries as David Sackett and Gordon Guyatt, among many others. EPC’s are brimming with experienced epidemiologists at both the PhD and masters degree level, as well as graduate students looking to make their mark working on clinically relevant epidemiology projects. They have the research staff and hardware and software infrastructure in place to support even the largest conceived projects. Overall, they are in a better position to provide the highest quality literature search, evidence sifting, and initial evidence table organization in the shortest possible time, than just about any professional medical society.

On the other hand, the medical professional societies contain the clinical subject matter experts that are necessary and critical for final review and assessment of the evidence in the evidence tables, creation of the subsequent clinical recommendations, and adjudication of the strength of recommendations created. Guidelines construction methodology and multidisciplinary panel composition are equally important in determining the overall quality and validity of the guideline produced. However, multidisciplinary panel composition is probably even more important than guidelines construction methodology in influencing subsequent guideline buy-in, acceptance, and implementation by clinicians for clinical care. An EBM clinical practice guideline unilaterally produced by EPC epidemiologists is far less likely to be adopted, regardless of its degree of methodological purity.

Areas that could be improved

The biggest drawback to continuing to follow this paradigm is the cost involved. This pilot project for EBM clinical practice guidelines cost the AANS, CNS, and the Joint Tumor Section ~$250,000 to produce. Given the number of areas of neurosurgical practice that are crying out for guidelines development coupled with the need to revise these guidelines approximately every 5 years, it becomes clear that the whole budget of these organizations could very rapidly become consumed with the need to produce and revise EBM clinical practice guidelines. Regardless of the altruistic intent of the medical organization or the strength of the desire to serve our patients and members, this duty would soon become economically unsustainable.

Given that EBM clinical practice guidelines have the strong potential to benefit hospitals, private insurance companies, government third party payers, the US government budget, patients, and society as a whole, a better means of addressing guidelines development cost would seem to be in order. The AHRQ is a division of the US Government under the Department of HHS. It is currently spending over $200,000,000 per year funding EPC’s. It would seem that this funding and granting arrangement could be revised to better align incentives to promote EPC and professional medical society collaboration and partnership, promote joint EBM clinical practice guideline development, and control costs to nonprofit professional medical societies who provide the expert panel writing group expertise. Specific suggestions include:

  1. (A)

    Making EPC collaboration with professional medical societies for the purpose of EBM clinical practice guideline development a condition for AHRQ grant funding and/or part of grant evaluation for success or continuance

  2. (B)

    Providing matching funds for professional medical societies to contract with AHRQ-funded EPCs for the purpose of EBM clinical practice guideline development

  3. (C)

    Create a condition for AHRQ EPC funding that would exempt professional medical societies form paying indirect costs when contracting with EPCs for the purpose of EBM clinical practice guideline development (indirect costs constituted 40% of the $250,000 spent on the current EBM clinical practice guideline effort).

Conclusion

In the absence of better cost sharing planning and regulations for EBM clinical practice guideline development, the synergistic advantages of EPC—professional medical society partnerships may not be cost effective or sustainable. Alternatives include individual professional medical societies hiring a minimum core of their own epidemiologists, researchers, and other support personnel and maintaining their own hardware and software infrastructure as fixed annual costs, to work with their members on these projects. This would be better than the worst case scenario of abandoning EBM clinical practice guideline efforts all-together, but would suffer from failing to leverage the existing professional expertise and infrastructure power of our 14 tax payer-funded EPCs.

The brain metastasis EBM clinical practice guideline presented in this special edition of the Journal of Neuro-Oncology is a powerful and exciting new addition for optimizing the quality of care for our patients with metastatic brain tumors. As a new AANS, CNS and Joint Tumor Section pilot project for exploring the feasibility and utility of contracting with an AHRQ-funded EPC for producing the highest quality EBM clinical practice guidelines in the shortest possible period of time, it has been a resounding success. The guidelines production paradigm is translatable to all areas of medicine and many other professional medical societies. paradigm Whether or not it becomes economically sustainable over time depends in large part on funding re-engineering and planning efforts outside professional medical society control, at the health policy and governmental level within such departments as the Department of HHS.