Abstract
Interest in the development and promotion of clinical practice guidelines (CPGs) continues to grow in many professions. However, the potential benefits associated with CPGs are dependent upon their quality. A number of studies have shown that the quality of CPGs varies greatly. Furthermore, the quality of many of the CPGs used in health and social sciences has yet to be examined. In light of this, the aim of this study was to examine the quality of CPGs that focus on intervention and care management in mental health in Quebec. A search of Quebec regulatory bodies websites was conducted and six CPGs were included in this study. The CPGs were assessed by four trained raters using the Appraisal of Guidelines for Research and Evaluation II (AGREE II) instrument. Items scores and domains scores were considered to determine the quality of the six CPGs. Results show that many of the CPGs did not achieve minimum ratings for numerous quality checks. Notably, none of the CPGs were designed using a rigorous methodology, they lacked transparency throughout the development process and insufficient consideration was given to the applicability of the recommendations they included. Because these shortcomings may hinder the efficacy and utilization of CPGs, suggestions to improve the development of CPGs and to improve their quality are discussed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Le médecin, la télémédecine et les technologies de l’information et de la communication (CMQ, 2015).
Lignes directrices sur le plan d’intervention pour une personne hébergée en centre jeunesse (OPPQ, 2014).
Guide de pratique: Orientation en formation générale des jeunes (OCCOQ, 2013).
Décider de l’utilisation des mesures de contention et d’isolement dans le cadre de la Loi sur les services de santé et des services sociaux et de la loi sur les services de santé et des services sociaux pour les autochtones cris (OTSTCFQ, 2011).
Déterminer un plan d’intervention pour une personne atteinte d’un trouble mental ou présentant un risque suicidaire qui est hébergée dans une installation d’un établissement qui exploite un centre de réadaptation pour les jeunes en difficulté d’adaptation (OTSTCFQ, 2013).
Interventions relatives à l’utilisation d’un véhicule routier: Guide de l’ergothérapeute (OEQ, 2008).
References
AGREE Collaboration. (2003). Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: The AGREE project. Quality and Safety in Health Care, 12(1), 18–23. https://doi.org/10.1136/qhc.12.1.18
Alonso-Coello, P., Irfan, A., Sola, I., Gich, I., Delgado-Noguera, M., Rigau, D., Tort, S., Bonfill, X., Burgers, J., & Schunemann, H. (2010). The quality of clinical practice guidelines over the last two decades: A systematic review of guideline appraisal studies. Quality and Safety in Health Care, 19(6), e58. https://doi.org/10.1136/qshc.2010.042077
Ansari, S., & Rashidian, A. (2012). Guidelines for guidelines: Are they up to the task? A comparative assessment of clinical practice guideline development handbooks. PLoS ONE, 7(11), e49864. https://doi.org/10.1371/journal.pone.0049864
Armstrong, J. J., Goldfarb, A. M., Instrum, R. S., & MacDermid, J. C. (2017). Improvement evident but still necessary in clinical practice guideline quality: A systematic review. Journal of Clinical Epidemiology, 81, 13–21. https://doi.org/10.1016/j.jclinepi.2016.08.005
Armstrong, J. J., Rodrigues, I. B., Wasiuta, T., & MacDermid, J. C. (2016). Quality assessment of osteoporosis clinical practice guidelines for physical activity and safe movement: An AGREE II appraisal. Archives of Osteoporosis, 11, 6. https://doi.org/10.1007/s11657-016-0260-9
Barriocanal, A. M., Lopez, A., Monreal, M., & Montane, E. (2016). Quality assessment of peripheral artery disease clinical guidelines. Journal of Vascular Surgery, 63(4), 1091–1098. https://doi.org/10.1016/j.jvs.2015.12.040
Beauchamp, S., Drapeau, M., & Dionne, C. (2015). The development of practice guidelines in the social and human sciences. Canadian Psychology/Psychologie canadienne, 56(4), 357–367. https://doi.org/10.1037/a0038811
Blozik, E., Nothacker, M., Bunk, T., Szecsenyi, J., Ollenschlager, G., & Scherer, M. (2012). Simultaneous development of guidelines and quality indicators—How do guideline groups act? A worldwide survey. International Journal of Health Care Quality Assurance, 25(8), 712–729. https://doi.org/10.1108/09526861211270659
Brouwers, M. C., Kho, M. E., Browman, G. P., Burgers, J. S., Cluzeau, F., Feder, G., Fervers, B., Graham, I. D., Grimshaw, J., Hanna, S. E., Littlejohns, P., & Zitzelsberger, L. (2010a). AGREE II: Advancing guideline development, reporting, and evaluation in health care. CMAJ, 51(5), 421–424. https://doi.org/10.1503/cmaj.090449
Brouwers, M. C., Kho, M. E., Browman, G. P., Burgers, J. S., Cluzeau, F., Feder, G., Graham, I. D., Hanna, S. E., & Makarski, J. (2010b). Development of the AGREE II, part 1: Performance, usefulness and areas for improvement. CMAJ, 182(10), 1045–1052. https://doi.org/10.1503/cmaj.091714
Burgers, J. S., Fervers, B., Haugh, M., Brouwers, M., Browman, G., Philip, T., & Cluzeau, F. A. (2004). International assessment of the quality of clinical practice guidelines in oncology using the Appraisal of Guidelines and Research and Evaluation Instrument. Journal of Clinical Oncology, 22(10), 2000–2007. https://doi.org/10.1200/jco.2004.06.157
Burgers, J. S., Grol, R., Klazinga, N. S., Makela, M., & Zaat, J. (2003). Towards evidence-based clinical practice: An international survey of 18 clinical guideline programs. International Journal for Quality in Health Care, 15(1), 31–45. https://doi.org/10.1093/intqhc/15.1.31
Burgers, J. S., Cluzeau, F. A., Hanna, S. E., Hunt, C., & Grol, R. (2003). Characteristics of high-quality guidelines: Evaluation of 86 clinical guidelines developed in ten European countries and Canada. International Journal of Technology Assessment in Health Care, 19(1), 148–157
Cates, J. R., Young, D. N., Bowerman, D. S., & Porter, R. C. (2006). An independent AGREE evaluation of the Occupational Medicine Practice Guidelines. The Spine Journal, 6(1), 72–77. https://doi.org/10.1016/j.spinee.2005.06.012
Chambless, D. L., & Ollendick, T. H. (2001). Empirically supported psychological interventions: Controversies and evidence. Annual Review of Psychology, 52, 685–716. https://doi.org/10.1146/annurev.psych.52.1.685
Chang, S. G., Kim, D. I., Shin, E. S., Jang, J. E., Yeon, J. Y., & Lee, Y. S. (2016). Methodological quality appraisal of 27 Korean Guidelines using a scoring guide based on the AGREE II Instrument and a Web-based Evaluation. Journal of Korean Medical Science, 31(5), 682–687. https://doi.org/10.3346/jkms.2016.31.5.682
Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290. https://doi.org/10.1037/1040-3590.6.4.284
Ciquier, G., Azzi, M., Hébert, C., Watkins-Martin, K., & Drapeau, M. (2021). Assessing the quality of seven clinical practice guidelines from four professional regulatory bodies in Quebec: What’s the verdict? Journal of Evaluation in Clinical Practice, 27(1), 25–33. https://doi.org/10.1111/jep.13374
CMQ. (2015). Le médecin, la télémedecine et les technologies de l’information et de la communication. Retrieved August 2018, from http://www.cmq.org/publications-pdf/p-1-2015-02-01-fr-medecin-telemedecine-et-tic.pdf
Dans, A. L. (2008). Applying the results to individual patients. In G. Guyatt & R. Drummond (Eds.), Users’ guide to the medical literature.American Medical Association.
Dans, A. L., & Dans, L. F. (2010). Appraising a tool for guideline appraisal (the AGREE II instrument). Journal of Clinical Epidemiology, 63(12), 1281–1282. https://doi.org/10.1016/j.jclinepi.2010.06.005
Diaz Del Campo, P., Gracia, J., Blasco, J. A., & Andradas, E. (2011). A strategy for patient involvement in clinical practice guidelines: Methodological approaches. BMJ Quality and Safety, 20(9), 779–784. https://doi.org/10.1136/bmjqs.2010.049031
Don-Wauchope, A. C., Sievenpiper, J. L., Hill, S. A., & Iorio, A. (2012). Applicability of the AGREE II instrument in evaluating the development process and quality of current National Academy of Clinical Biochemistry guidelines. Clinical Chemistry, 58(10), 1426–1437. https://doi.org/10.1373/clinchem.2012.185850
Dozois, D. J., Mikail, S. F., Alden, L. E., Bieling, P. J., Bourgon, G., Clark, D. A., Drapeau, M., Gallson, D., Greenberg, L., Hunsley, J., & Johnston, C. (2014). The CPA Presidential task force on evidence-based practice of psychological treatments. Canadian Psychology, 55(3), 153–160. https://doi.org/10.1037/a0035767
Drapeau, M., & Hunsley, J. (2014). Where’s the science? Introduction to a special issue of Canadian Psychology on science in psychology. Canadian Psychology/Psychologie canadienne, 55(3), 145–152. https://doi.org/10.1037/a0037321
Graham, I. D., Mancher, M., Wolman, D. M., Greenfield, S., & Steinberg, E. (2011). Clinical practice guidelines we can trust. National Academies Press.
Graham, I. D., Beardall, S., Carter, A. O., Tetroe, J., & Davies, B. (2003). The state of the science and art of practice guidelines development, dissemination and evaluation in Canada. Journal of Evaluation in Clinical Practice, 9(2), 195–202. https://doi.org/10.1046/j.1365-2753.2003.00385.x
Graham, I. D., Calder, L. A., Hebert, P. C., Carter, A. O., & Tetroe, J. M. (2000). A comparison of clinical practice guideline appraisal instruments. International Journal of Technology Assessment in Health Care, 16(4), 1024–1038
Hollon, S. D., Arean, P. A., Craske, M. G., Crawford, K. A., Kivlahan, D. R., Magnavita, J. J., Ollendick, T. H., Sexton, T. L., Spring, B., Bufka, L. F., Galper, D. I., & Kurtzman, H. (2014). Development of clinical practice guidelines. Annual Review of Clinical Psychology, 10, 213–241. https://doi.org/10.1146/annurev-clinpsy-050212-185529
Holmer, H. K., Ogden, L. A., Burda, B. U., & Norris, S. L. (2013). Quality of clinical practice guidelines for glycemic control in type 2 diabetes mellitus. PLoS ONE, 8(4), e58625. https://doi.org/10.1371/journal.pone.0058625
Kazdin, A. E. (2008). Evidence-based treatment and practice: New opportunities to bridge clinical research and practice, enhance the knowledge base, and improve patient care. American Psychologist, 63(3), 146–159. https://doi.org/10.1037/0003-066X.63.3.146
Kredo, T., Bernhardsson, S., Machingaidze, S., Young, T., Louw, Q., Ochodo, E., & Grimmer, K. (2016). Guide to clinical practice guidelines: The current state of play. International Journal for Quality in Health Care, 28(1), 122–128. https://doi.org/10.1093/intqhc/mzv115
Kung, J., Miller, R. R., & Mackowiak, P. A. (2012). Failure of clinical practice guidelines to meet institute of medicine standards: Two more decades of little, if any, progress. Archives of internal medicine, 172(21), 1628–1633. https://doi.org/10.1001/2013.jamainternmed.56
MacQueen, G., Santaguida, P., Keshavarz, H., Jaworska, N., Levine, M., Beyene, J., & Raina, P. (2017). Systematic review of clinical practice guidelines for failed antidepressant treatment response in major depressive disorder, dysthymia, and subthreshold depression in adults. The Canadian Journal of Psychiatry, 62(1), 11–23. https://doi.org/10.1177/0706743716664885
Marshall, S., Bayley, M., McCullagh, S., Velikonja, D., Berrigan, L., Ouchterlony, D., & Weegar, K. (2015). Updated clinical practice guidelines for concussion/mild traumatic brain injury and persistent symptoms. Brain Injury, 29(6), 688–700. https://doi.org/10.3109/02699052.2015.1004755
Medina, J. C., Schmelefske, E., Hébert, C., & Drapeau, M. (2020). European clinical practice guidelines for depression in adults: Are they good enough? Journal of Affective Disorders, 263, 382–385. https://doi.org/10.1016/j.jad.2019.12.005
Middleton, J. C., Kalogeropoulos, C., Middleton, J. A., & Drapeau, M. (2018). Assessing the methodological quality of the Canadian Psychiatric Association’s anxiety and depression clinical practice guidelines. Journal of Evaluation in Clinical Practice. https://doi.org/10.1111/jep.13026
Montori, V. M., Brito, J. P., & Murad, M. H. (2013). The optimal practice of evidence-based medicine: Incorporating patient preferences in practice guidelines. JAMA, 310(23), 2503–2504. https://doi.org/10.1001/jama.2013.281422
Murad, M. H. (2017). Clinical practice guidelines: A primer on development and dissemination. In K. A. Nath (Ed.), Mayo Clinic Proceedings. (Vol. 92, pp. 423–433). Elsevier. https://doi.org/10.1016/j.mayocp.2017.01.001
Norris, S. L., Holmer, H. K., Ogden, L. A., & Burda, B. U. (2011). Conflict of interest in clinical practice guideline development: A systematic review. PLoS ONE, 6(10), e25153. https://doi.org/10.1371/journal.pone.0025153
OCCOQ. (2013). Guide de pratique: Orientation en formation générale des jeunes. Retrieved August 2018, from https://orientation.qc.ca/files/Guide_OCCOQ_OFGJ_250214.pdf
OEQ. (2008). Interventions relatives à l’utilisation d’un véhicule routier: Guide de l’ergothérapeute. Retrieved August 2018, from https://www.oeq.org/DATA/NORME/30~v~guide_auto.pdf
OPPQ. (2014). Déterminer un plan d’intervention pour une personne atteinte d’un trouble mental ou présentant un risque suicidaire qui est hébergée dans une installation d’un établissement qui exploite un centre de réadaptation pour les jeunes en difficulté d’adaptation. Retrieved August 2018, from http://www.ordrepsed.qc.ca/~/media/pdf/Publication/LignesdirectricesDterminerunPI.ashx?la=fr
OTSTCFQ. (2011). Lignes directrices: Décider de l’utilisation des mesures de contention et d’isolement dans le cadre de la Loi sur les services de santé et des services sociaux et de la loi sur les services de santé et des services sociaux pour les autochtones cris. Retrieved August 2018, from https://beta.otstcfq.org/sites/default/files/decider-de-l-utilisation-des-mesures-de-contention-et-d-isolement.pdf
OTSTCFQ. (2013). Lignes directrices: Déterminer un plan d’intervention pour une personne atteinte d’un trouble mental ou présentant un risque suicidaire qui est hébergée dans une installation d’un établissement qui exploite un centre de réadaptation pour les jeunes en difficulté d’adaptation. Retrieved August 2018, from https://beta.otstcfq.org/sites/default/files/determiner_un_plan_dintervention_pour_une_personne_atteinte_dun_trouble_mental_ou_presentant_un_risque_suicidaire.pdf
Oxman, A. D., Schunemann, H. J., & Fretheim, A. (2006). Improving the use of research evidence in guideline development: 16. Evaluation. Health Research Policy and Systems, 4, 28. https://doi.org/10.1186/1478-4505-4-28
Parry, G., Cape, J., & Pilling, S. (2003). Clinical practice guidelines in clinical psychology and psychotherapy. Clinical Psychology & Psychotherapy, 10(6), 337–351. https://doi.org/10.1002/cpp.381
Rosenfeld, R. M., Shiffman, R. N., & Robertson, P. (2013). Clinical practice guideline development manual, Third Edition: A quality-driven approach for translating evidence into action. Otolaryngology-Head and Neck Surgery, 148(1_suppl), S1–S55. https://doi.org/10.1177/0194599812467004
Sabharwal, S., Patel, N. K., Gauher, S., Holloway, I., & Athanasiou, T. (2014). High methodologic quality but poor applicability: Assessment of the AAOS guidelines using the AGREE II instrument. Clinical Orthopaedics and Related Research, 472(6), 1982–1988. https://doi.org/10.1007/s11999-014-3657-z
Shekelle, P. G. (2014). Updating practice guidelines. JAMA, 311(20), 2072–2073. https://doi.org/10.1001/jama.2014.4950
Stamoulos, C., Reyes, A., Trepanier, L., & Drapeau, M. (2014). The methodological quality of clinical practice guidelines in psychology: A pilot study of the guidelines developed by the Ordre des Psychologues du Québec. Canadian Psychology/Psychologie canadienne, 55(3), 177. https://doi.org/10.1037/a0037287
Trepanier, L., Stamoulos, C., & Reyes, A. (2017). Assessing the quality of the OPQ’s guidelines for the evaluation of dyslexia in children using the appraisal of guidelines for research and evaluation (AGREE) II: A brief report. Canadian Psychology/Psychologie canadienne, 58(3), 211. https://doi.org/10.1037/cap0000121
Weisz, G., Cambrosio, A., Keating, P., Knaapen, L., Schlich, T., & Tournay, V. J. (2007). The emergence of clinical practice guidelines. Milbank Quarterly, 85(4), 691–727. https://doi.org/10.1111/j.1468-0009.2007.00505.x
WHO. (2012). Handbook for Guideline Development. Retrieved in August 2019 from https://apps.who.int/iris/bitstream/handle/10665/75146/9789241548441_eng.pdf?sequence=1
Woolf, S., Schunemann, H. J., Eccles, M. P., Grimshaw, J. M., & Shekelle, P. (2012). Developing clinical practice guidelines: Types of evidence and outcomes; values and economics, synthesis, grading, and presentation and deriving recommendations. Implementation Science, 7, 61. https://doi.org/10.1186/1748-5908-7-61
Woolf, S. H., Grol, R., Hutchinson, A., Eccles, M., & Grimshaw, J. (1999). Clinical guidelines: Potential benefits, limitations, and harms of clinical guidelines. BMJ, 318(7182), 527–530. https://doi.org/10.1136/bmj.318.7182.527
Yaşar, I., Kahveci, R., Baydar Artantaş, A., Ayhan Başer, D., Gökşin Cihan, F., Şencan, I., Koç, E. M., & Özkara, A. (2016). Quality Assessment of Clinical Practice Guidelines Developed by Professional Societies in Turkey. PLoS ONE, 11(6), e0156483. https://doi.org/10.1371/journal.pone.0156483
Ye, Z.-K., Liu, Y., Cui, X.-L., & Liu, L.-H. (2016). Critical appraisal of the quality of clinical practice guidelines for stress ulcer prophylaxis. PLoS ONE, 11(5), e0155020. https://doi.org/10.1371/journal.pone.0155020
Zafra-Tanaka, J. H., Goicochea-Lugo, S., Villarreal-Zegarra, D., & Taype-Rondan, A. (2019). Characteristics and quality of clinical practice guidelines for depression in adults: A scoping review. BMC Psychiatry, 19(1), 76. https://doi.org/10.1186/s12888-019-2057-z
Funding
This work was supported by the Social Sciences and Humanities Research Council (SSHRC) under Grant [435–2015], to [MD]. For this study, we used the methodological guidelines elaborated by the AGREE Consortium to examine the quality of the clinical practice guidelines included in this study (please refer to following references found in the manuscript: AGREE Collaboration, 2003; Brouwers et al., 2010a, b).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors have no conflict of interest to disclose.
Ethical Approval
Ethical approval for this study was not necessary.
Research Involving Human and Animal Rights
This study did not involve human participants or animals.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Hébert, C., Watkins-Martin, K., Ciquier, G. et al. The Quality of Six Clinical Practice Guidelines in Health and Social Sciences: Are We on the Right Track?. Adm Policy Ment Health 48, 821–829 (2021). https://doi.org/10.1007/s10488-021-01132-8
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-021-01132-8