Skip to main content

Rigorous Development does not Ensure that Guidelines are Acceptable to a Panel of Knowledgeable Providers

Journal of General Internal Medicine volume 23, pages 37–44 (2008)Cite this article

Abstract

BACKGROUND

Rigorous guideline development methods are designed to produce recommendations that are relevant to common clinical situations and consistent with evidence and expert understanding, thereby promoting guidelines’ acceptability to providers. No studies have examined whether this technical quality consistently leads to acceptability.

OBJECTIVE

To examine the clinical acceptability of guidelines having excellent technical quality.

DESIGN AND MEASUREMENTS

We selected guidelines covering several musculoskeletal disorders and meeting 5 basic technical quality criteria, then used the widely accepted AGREE Instrument to evaluate technical quality. Adapting an established modified Delphi method, we assembled a multidisciplinary panel of providers recommended by their specialty societies as leaders in the field. Panelists rated acceptability, including “perceived comprehensiveness” (perceived relevance to common clinical situations) and “perceived validity” (consistency with their understanding of existing evidence and opinions), for ten common condition/therapy pairs pertaining to Surgery, physical therapy, and chiropractic manipulation for lumbar spine, shoulder, and carpal tunnel disorders.

RESULTS

Five guidelines met selection criteria. Their AGREE scores were generally high indicating excellent technical quality. However, panelists found 4 guidelines to be only moderately comprehensive and valid, and a fifth guideline to be invalid overall. Of the topics covered by each guideline, panelists rated 50% to 69% as “comprehensive” and 6% to 50% as “valid”.

CONCLUSION

Despite very rigorous development methods compared with guidelines assessed in prior studies, experts felt that these guidelines omitted common clinical situations and contained much content of uncertain validity. Guideline acceptability should be independently and formally evaluated before dissemination.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

References

  1. Mosca L, Linfante AH, Benjamin EJ, et al. National study of physician awareness and adherence to cardiovascular disease prevention guidelines. Circulation. 2005;111(4):499–510.

    PubMed  Article  Google Scholar 

  2. Bauer MS. A review of quantitative studies of adherence to mental health clinical practice guidelines. Harv Rev Psychiatry. 2002;10(3):138–53.

    PubMed  Article  Google Scholar 

  3. Cruz-Correa M, Gross CP, Canto MI, et al. The impact of practice guidelines in the management of Barrett esophagus: a national prospective cohort study of physicians. Arch Intern Med. 2001;161(21):2588–95.

    PubMed  Article  CAS  Google Scholar 

  4. Switzer GE, Halm EA, Chang CC, Mittman BS, Walsh MB, Fine MJ. Physician awareness and self-reported use of local and national guidelines for community-acquired pneumonia. J Gen Intern Med. 2003;18(10):816–23.

    PubMed  Article  Google Scholar 

  5. Christakis DA, Rivara FP. Pediatricians’ awareness of and attitudes about four clinical practice guidelines. Pediatrics. 1998;101(5):825–30.

    PubMed  Article  CAS  Google Scholar 

  6. Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999;282(15):1458–65.

    PubMed  Article  CAS  Google Scholar 

  7. Ward MM, Vaughn TE, Uden-Holman T, Doebbeling BN, Clarke WR, Woolson RF. Physician knowledge, attitudes and practices regarding a widely implemented guideline. J Eval Clin Pract. 2002;8(2):155–62.

    PubMed  Article  Google Scholar 

  8. Cabana MD, Ebel BE, Cooper-Patrick L, Powe NR, Rubin HR, Rand CS. Barriers pediatricians face when using asthma practice guidelines. Arch Pediatr Adolesc Med. 2000;154(7):685–93.

    PubMed  CAS  Google Scholar 

  9. Committee to Advise the Public Health Service on Clinical Practice Guidelines, Institute of Medicine. In: Field MJ, Lohr KN, eds. Clinical Practice Guidelines: Directions for a New Program. Washington, DC: National Academies Press; 1990.

    Google Scholar 

  10. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J. Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA. 1999;281(20):1900–5.

    PubMed  Article  CAS  Google Scholar 

  11. The AGREE Collaboration. The Appraisal of Guidelines for Research & Evaluation (AGREE) Instrument. Available at: http://www.agreetrust.org/docs/AGREE_Instrument_English.pdf. Accessed March 21, 2007.

  12. The AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003;12(1):18–23.

    Article  Google Scholar 

  13. Burgers JS. Guideline quality and guideline content: are they related? Clin Chem. 2006;52(1):3–4.

    PubMed  Article  CAS  Google Scholar 

  14. Shiffman RN, Shekelle P, Overhage JM, Slutsky J, Grimshaw J, Deshpande AM. Standardized reporting of clinical practice guidelines: a proposal from the Conference on Guideline Standardization. Ann Intern Med. 2003;139(6):493–8.

    PubMed  Google Scholar 

  15. United States National Guideline Clearinghouse Inclusion Criteria. Available at: http://www.guideline.gov/about/inclusion.aspx. Accessed July 6, 2006.

  16. State of California. Assembly Bill 749 (Calderon), Workers’ Compensation: Administration and Benefits; 2002.

  17. State of California. Senate Bill 228 (Alarcón), Workers’ Compensation; 2003.

  18. State of California. Senate Bill 899 (Poochigan), Workers’ Compensation; 2004.

  19. Hutchings A, Raine R, Sanderson C, Black N. An experimental study of determinants of the extent of disagreement within clinical guideline development groups. Qual Saf Health Care. 2005;14(4):240–5.

    PubMed  Article  CAS  Google Scholar 

  20. Nuckols TK, Wynn BO, Lim Y, et al. Evaluating Medical Treatment Guideline Sets for Injured Workers in California. Santa Monica, CA: RAND Corporation; 2005. MG-400-ICJ.

  21. Grol R, Cluzeau FA, Burgers JS. Clinical practice guidelines: towards better quality guidelines and increased international collaboration. Br J Cancer. 2003;89(Suppl):S4–8.

    PubMed  Article  Google Scholar 

  22. Watine J, Friedberg B, Nagy E, et al. Conflict between guideline methodologic quality and recommendation validity: a potential problem for practitioners. Clin Chem. 2006;52(1):65–72.

    PubMed  Article  CAS  Google Scholar 

  23. Vlayen J, Aertgeerts B, Hannes K, Sermeus W, Ramaekers D. A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. Int J Qual Health Care. 2005;17(3):235–42.

    PubMed  Article  Google Scholar 

  24. The AGREE Collaboration. The Appraisal of Guidelines for Research & Evaluation (AGREE) Instrument Training Manual. Available at: http://www.agreetrust.org/docs/AGREE_INSTR_MANUAL.pdf. Accessed March 22, 2007.

  25. Fitch K, Bernstein SJ, Aguilar MD, et al. The RAND/UCLA Appropriateness Method User’s Manual. Santa Monica, CA: RAND Corporation; 2001. MR-1269-DG-XII/RE. Available at: http://www.rand.org/publications/MR/MR1269/.

  26. Shekelle PG. The appropriateness method. Med Decis Making. 2004;24(2):228–31.

    PubMed  Article  Google Scholar 

  27. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74.

    PubMed  Article  CAS  Google Scholar 

  28. Shekelle PG, Kahan JP, Bernstein SJ, Leape LL, Kamberg CJ, Park RE. The reproducibility of a method to identify the overuse and underuse of medical procedures. N Engl J Med. 1998;338(26):1888–95.

    PubMed  Article  CAS  Google Scholar 

  29. Shekelle PG, Chassin MR, Park RE. Assessing the predictive validity of the RAND/UCLA appropriateness method criteria for performing carotid endarterectomy. Int J Technol Assess Health Care. 1998;14(4):707–27.

    PubMed  CAS  Article  Google Scholar 

  30. Higashi T, Shekelle PG, Adams JL, Kamberg CJ, Roth CP, Solomon DH, Reuben DB, Chiang L, MacLean CH, Chang JT, Young RT, Saliba DM, Wenger NS. Quality of care is associated with survival in vulnerable older patients. Ann Intern Med. 2005 Aug 16;143(4):274–81.

    PubMed  Google Scholar 

  31. Wynn BO, Bergamo G, Shaw RN, Mattke S, Dembe A. Medical Care Provided California’s Injured Workers: An Overview of the Issues. Santa Monica, CA: RAND Corporation; 2006. WR-394-ICJ.

  32. Hasenfeld R, Shekelle PG. Is the methodological quality of guidelines declining in the U.S.? Comparison of the quality of U.S. agency for health care policy and research (AHCPR) guidelines with those published subsequently. Qual Saf Health Care. 2003;12(6):428–34.

    PubMed  Article  CAS  Google Scholar 

  33. Brook RH, McGlynn EA, Cleary PD. Quality of health care. Part 2: measuring quality of care. N Engl J Med. 1996;335(13):966–70.

    PubMed  Article  CAS  Google Scholar 

  34. Donabedian A. Explorations in Quality Assessment and Monitoring, Vol. 1. The Definition of Quality and Approaches to its Assessment. Ann Arbor, MI: Health Administration Press; 1980.

  35. Donabedian A. Explorations in Quality Assessment and Monitoring, Vol. 2. The Criteria and Standards of Quality. Ann Arbor, MI: Health Administration Press; 1982.

  36. Donabedian A. Explorations in Quality Assessment and Monitoring, Vol. 3. The Methods and Findings of Quality Assessment and Monitoring: An Illustrated Analysis. Ann Arbor, MI: Health Administration Press; 1985.

  37. Shekelle PG, Woolf SH, Eccles M, Grimshaw J. Clinical guidelines: developing guidelines. BMJ. 1999;318(7183):593–6.

    PubMed  CAS  Google Scholar 

  38. Maue SK, Segal R, Kimberlin CL, Lipowski EE. Predicting physician guideline compliance: an assessment of motivators and perceived barriers. Am J Manag Care. 2004;10(6):383–91.

    PubMed  Google Scholar 

  39. Mottur-Pilson C, Snow V, Bartlett K. Physician explanations for failing to comply with “best practices”. Eff Clin Pract. 2001;4(5):207–13.

    PubMed  CAS  Google Scholar 

  40. Grol R, Dalhuijsen J, Thomas S, Veld C, Rutten G, Mokkink H. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ. 1998;317(7162):858–61.

    PubMed  CAS  Google Scholar 

  41. Burgers JS, Grol RP, Zaat JO, Spies TH, van der Bij AK, Mokkink HG. Characteristics of effective clinical guidelines for general practice. Br J Gen Pract. 2003;53(486):15–9.

    PubMed  Google Scholar 

  42. Shiffman RN, Dixon J, Brandt C, et al. The GuideLine Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation. BMC Med Inform Decis Mak. 2005;5:23–34.

    PubMed  Article  Google Scholar 

  43. Shekelle PG, Ortiz E, Rhodes S, et al. Validity of the Agency for Healthcare Research and Quality Clinical Practice Guidelines: how quickly do guidelines become outdated? JAMA. 2001;286(12):1461–7.

    PubMed  Article  CAS  Google Scholar 

  44. American Academy of Orthopaedic Surgeons. Guidelines and support documents for hip pain, knee injury, knee osteoarthritis, low back pain/sciatica, shoulder pain and wrist pain. 2001–2003.

  45. American College of Occupational and Environmental Medicine. Occupational Medicine Practice Guidelines: Evaluation and Management of Common Health Problems and Functional Recovery in Workers, Second Edition. Beverly Farms, Massachusetts; 2004.

  46. Intracorp. Intracorp Optimal Treatment Guidelines. Philadelphia, Pennsylvania; 2003. Available at: http://www.intracorp.com.

  47. McKesson. InterQual Care Management Criteria for Workers’ Comp & Clinical Evidence Summaries. (Formerly QualityFirst® Workers’ Compensation/Disability Guidelines). Newton, Massachusetts. 2004.

  48. Work Loss Data Institute. Official Disability Guides (ODG), Treatment in Workers’ Compensation. San Diego, CA; 2004. Available at: http://www.worklossdata.com.

Download references

Acknowledgements

Christine Baker, M.A., Executive Officer of the California Commission on Health and Safety and Workers’ Compensation and Anne Searcy, M.D., Medical Director of the California Division of Workers’ Compensation participated in the study conception and design (specifically, choosing the inclusion criteria for the guidelines). Paul G. Shekelle, M.D, Ph.D., The RAND Corporation, Santa Monica; Division of General Internal Medicine and Health Services Research, Department of Medicine David Geffen School of Medicine at the University of California, Los Angeles; and VA Greater Los Angeles Healthcare System, Los Angeles, California, provided comments on a draft of this manuscript. This study was sponsored by the California Department of Industrial Relations (the Division of Workers Compensation and the California Commission on Health and Safety and Workers’ Compensation). The RAND Corporation provided support for the preparation of this manuscript. Although Christine Baker and Anne Searcy, Division of Workers’ Compensation participated in the study conception and design, data acquisition, analysis, and preparation of this manuscript were completely independent of the funders.

Conflicts of Interest

None disclosed.

Author information

Authors and Affiliations

  1. RAND Corporation, Santa Monica, CA, USA

    Teryl K. Nuckols MD, MSHS, Yee-Wei Lim MD, PhD, Barbara O. Wynn MA, Soeren Mattke MD, DSc, Robert H. Brook MD, ScD, Rena H. Garland & Steven Asch MD, MPH

  2. Division of General Internal Medicine and Health Services Research, Department of Medicine, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, USA

    Teryl K. Nuckols MD, MSHS, Robert H. Brook MD, ScD & Steven Asch MD, MPH

  3. WellPoint Incorporated, Thousand Oaks, CA, USA

    Catherine H. MacLean MD, PhD

  4. Division of Occupational and Environmental Medicine, Department of Family Medicine, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, USA

    Philip Harber MD, MPH

  5. ACS Healthcare Solutions Incorporated, Playa del Rey, CA, USA

    Peggy Wallace RN, MA, MS

  6. VA Greater Los Angeles Healthcare System, Los Angles, CA, USA

    Steven Asch MD, MPH

Authors
  1. Teryl K. Nuckols MD, MSHS
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yee-Wei Lim MD, PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Barbara O. Wynn MA
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Soeren Mattke MD, DSc
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Catherine H. MacLean MD, PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Philip Harber MD, MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Robert H. Brook MD, ScD
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Peggy Wallace RN, MA, MS
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Rena H. Garland
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Steven Asch MD, MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Teryl K. Nuckols MD, MSHS.

Additional information

This document includes content previously published in a RAND report, available online at: http://www.rand.org/publications/MG/MG400/.

Appendix

Appendix

Table 6 and 7

Table 6 Panel Judgments Regarding Perceived Comprehensiveness for 16 Selected Topics, Any Residual Content Aside from Those 16 Topics, and Overall (C = Comprehensive, I = Intermediate, NC = Not Comprehensive)
Full size table
Table 7 Panel Judgments Regarding Perceived Validity for 16 Selected Topics, Any Residual Content Aside from Those 16 Topics, and Overall
Full size table

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Nuckols, T.K., Lim, YW., Wynn, B.O. et al. Rigorous Development does not Ensure that Guidelines are Acceptable to a Panel of Knowledgeable Providers. J GEN INTERN MED 23, 37–44 (2008). https://doi.org/10.1007/s11606-007-0440-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11606-007-0440-9

KEY WORDS

  • practice guidelines
  • quality assurance
  • health care
  • evidence-based medicine
  • attitude of health personnel
  • evaluation