Abstract
Background
The KT Challenge program supports health care professionals to effectively implement evidence-based practices. Unlike other knowledge translation (KT) programs, this program is grounded in capacity building, focuses on health care professionals (HCPs), and uses a multi-component intervention. This study presents the evaluation of the KT Challenge program to assess the impact on uptake, KT capacity, and practice change.
Methods
The evaluation used a mixed-methods retrospective pre-post design involving surveys and review of documents such as teams’ final reports. Online surveys collecting both quantitative and qualitative data were deployed at four time points (after both workshops, 6 months into implementation, and at the end of the 2-year funded projects) to measure KT capacity (knowledge, skills, and confidence) and impact on practice change. Qualitative data was analyzed using a general inductive approach and quantitative data was analyzed using non-parametric statistics.
Results
Participants reported statistically significant increases in knowledge and confidence across both workshops, at the 6-month mark of their projects, and at the end of their projects. In addition, at the 6-month check-in, practitioners reported statistically significant improvements in their ability to implement practice changes. In the first cohort of the program, of the teams who were able to complete their projects, half were able to show demonstrable practice changes.
Conclusions
The KT Challenge was successful in improving the capacity of HCPs to implement evidence-based practice changes and has begun to show demonstrable improvements in a number of practice areas. The program is relevant to a variety of HCPs working in diverse practice settings and is relatively inexpensive to implement. Like all practice improvement programs in health care settings, a number of challenges emerged stemming from the high turnover of staff and the limited capacity of some practitioners to take on anything beyond direct patient care. Efforts to address these challenges have been added to subsequent cohorts of the program and ongoing evaluation will examine if they are successful. The KT Challenge program has continued to garner great interest among practitioners, even in the midst of dealing with the COVID-19 pandemic, and shows promise for organizations looking for better ways to mobilize knowledge to improve patient care and empower staff. This study contributes to the implementation science literature by providing a description and evaluation of a new model for embedding KT practice skills in health care settings.
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Background
There is an urgent need to reduce the gap between the creation of evidence and its implementation into practice. However, alarmingly, up to 70% of all organizational improvement efforts fail, including those in health care settings [1]. To address this shortcoming, knowledge translation (KT) initiatives and training programs in health care have proliferated in recent years [2,3,4,5,6,7]. Key barriers to successful KT include lack of knowledge, skills, time, and leaders’ endorsement, and competing priorities of organizations [8,9,10]. Important enablers include health care professionals’ (HCPs) positive beliefs about the benefits of participation in training programs, combined with expert guidance and organizational support [11]. HCPs want to provide the best care possible, but many lack the knowledge and skills to effectively move evidence into practice. A 2012 survey of health care providers, administrators, and researchers revealed that nearly 80% of respondents wanted to improve their knowledge and skills related to implementation [10]. Moreover, research has demonstrated that high-quality implementation strategies are associated with successful implementation [11], thus supporting the need to offer KT and implementation capacity building, training, and support to clinicians. A recent systematic review stressed the importance of increasing the number and reach of training opportunities to address the lack of dissemination and implementation training options [12].
Providing KT training takes a dedicated effort from health care organizations. Much of the research conducted on how to close the evidence-to-practice gap has been led by and/or focused on academic researchers with minimal involvement from those who provide or use health services [3]. First-generation interventions used co-creation models. For example, a program in the UK used a partnership approach to KT and reported that implementation projects co-produced by researchers and end users were more likely to successfully adapt new evidence into practice, and the co-production experience encouraged future collaboration between the parties [3]. Researchers and health care leaders have called for training that builds capacity among those working at the point of care [4,5,6], who often have the responsibility to implement, but do not have adequate skills and support to do so effectively. Involving those who use research knowledge in efforts to implement it has multiple benefits, including enhancing the effectiveness of implementation efforts [3]. Eames and colleagues, for example, found changes in clinician-reported behavior, especially the use of strategies for implementing a change in practice, following participation in their KT capacity-building intervention for occupational therapists, as well as changes in the culture to one in which clinicians engaged in KT as part of their clinical practice [2]. Similarly, a longitudinal evaluation of a program for implementers, called Practicing KT, showed increased use of, knowledge of, and self-efficacy in KT among those who completed the program [9].
Many programs exist to increase KT capacity, including some that offer training in implementation skills, but there is lack of information on their effectiveness. Programs that use active forms of learning to promote the acquisition of KT skills have been shown to be effective in transferring knowledge, but there is scant evidence of sustained practice change resulting from these sessions [7,8,9, 13]. KT scholars have acknowledged the need for longitudinal evaluation of KT capacity-building initiatives to assess the sustainability of outcomes from these programs [4, 9].
Armed with these findings, we developed an implementation support program for HCPs, aimed at building capacity to move research evidence into practice. The program, called the “KT Challenge,” was modeled on a similar research capacity-building initiative for clinicians. That program, in place at our organizations for more than 10 years, has been shown to be effective in supporting practice changes, enhancing evidence-based practice, and increasing interest in research engagement, including increased interest in graduate school [14, 15]. The KT Challenge program was implemented at two health organizations in British Columbia, Canada, and offered to all HCPs employed at the organizations. We report an evaluation of the KT Challenge program with respect to uptake by HCPs, impact on KT capacity (knowledge, skills, and confidence) of HCPs, and impact on evidence-based practice changes.
Methods
Overview of the KT Challenge program
The KT Challenge is a multi-component implementation support program that involves training, funding, peer review, and mentorship. The key components of the KT Challenge program are briefly described below.
Letter of Intent (LOI)
In the LOI, teams identify the practice change they want to implement, document the need for this change in their practice context, and summarize evidence of its effectiveness. LOIs also require the signature of a manager to ensure management support and endorsement, and the identification of team members. The LOIs are formatively reviewed and, in keeping with the capacity-building approach of this program, revisions are suggested when required. Some teams are screened out of the program at this point if their proposed LOI does not identify an evidence-based practice change, but most teams are given feedback and invited into the next stage.
Workshops
Teams attend two half-day workshops, focusing on developing an implementation plan, and evaluating the effectiveness of the implementation of the practice change. Topics include implementation theories and frameworks, stakeholder engagement, identifying barriers and facilitators, implementation strategies, and evaluation planning.
Mentorship
Teams are supported to find a mentor within their clinical area to support the implementation of the practice change and assist with navigating facilitators and barriers. Mentors are also invited to participate in the evaluation of the KT Challenge program throughout the project.
Online resources
Teams have access to an online learning site where a curated set of readings and resources is posted.
Management support
Teams are required to obtain their manager’s support for conducting the implementation project and dedicating time to it.
Funding
Successful teams are awarded $5000 to cover costs related to personnel, materials and supplies, stakeholder engagement, and services.
Program leads
The KT Challenge is run by a program lead from each organization. The leads promote the program, coordinate the review processes, monitor team progress, support teams to successfully navigate barriers, and lead the evaluation of the program.
Figure 1 outlines the timeline of activities for each cohort. Funded teams are asked to submit a quarterly report to provide an update on their progress, as well as a final report on the impact of their project on the intended practice changes.
Evaluation plan
To evaluate the KT Challenge, we used a mixed-methods retrospective pre-post design [15, 16] to assess the impact of the program on workshop participants and funded team members. The evaluation focused on the program uptake and changes in KT capacity (knowledge, skills, and confidence) among HCPs, and the impact on practice change. Online surveys (including both Likert scale and open-ended response questions) were used to collect data at four time points: after each of the two workshops, 6 months into implementation, and at the end of their 2-year funding. Only team leaders and mentors of funded teams complete the 6-month and end-of-project surveys. The online surveys collect both quantitative and qualitative data on participants’ knowledge, confidence, and ability. The 6-month check-in survey assessed knowledge, confidence, and ability by asking participants to rate their (1) overall knowledge of how to support practice changes, (2) level of confidence in supporting practice changes, and (3) ability to support practice changes on a 5-point scale that ranged from low (1) to high (5) before the program and 6 months into their projects. In this paper, we only consider data from the 6-month check-in survey because it captures a robust operationalization of KT capacity (knowledge, confidence, and ability) during the actual implementation of the practice changes and offers a larger sample size than the end-of-project survey which had only been completed by one cohort. Data on the impact of the practice change is gleaned from the final reports submitted by the team leads.
Data analysis
The qualitative data were analyzed using a general inductive approach, which involved the identification of themes and sub-themes related to the quality of the program, usefulness of various program components, impact on participants, impact on practice changes, and challenges. The themes were initially identified by one of the authors and then reviewed by two other authors. The quantitative data were analyzed in Excel (Microsoft Corporation), using the Wilcoxon signed-ranks test to compare pre- and post-scores on self-ratings of knowledge, skills, confidence, and ability. Statistical significance was set at p < 0.05.
Results
Uptake of the KT Challenge by practitioners
Four cohorts of participants have taken part in the program, with a fifth cohort recently launched. The first cohort (2016) has completed their projects. To date, 24 teams have been funded, comprising 185 HCPs. Participants have included a wide range of HCPs involving 23 types of practitioners working within a range of practice settings (see Fig. 2).
KT Challenge program participants by profession. Program participants included nurses, physicians, nurse-practitioners, allied health, and others. The “other” category included peer outreach workers, Aboriginal patient navigators, youth care workers, research scientists, overdose prevention specialists, etc.
Interest in the program has remained steady, with 13 teams participating in cohort 3 (6 were funded). While 9 teams initially joined cohort 4, the number of teams dropped significantly when the COVID-19 pandemic struck, and only 2 teams submitted funding proposals. However, cohort 5 was launched in October 2020, and 13 teams applied to participate, including teams proposing COVID-related projects.
KT capacity
The average survey response rate from the three cohorts was 76% for post-workshop #1 survey, 59% for post-workshop #2 survey, and 100% for both the 6-month check-in survey and the end-of-project survey.
Participants reported statistically significant increases in knowledge and confidence at the 6-month mark of their projects, compared with before the initiative (p < 0.05). In addition, at the 6-month check-in, practitioners reported statistically significant increases in their ability to implement practice changes (Fig. 3).
Numerous comments were provided on the surveys and in the final reports attesting to the knowledge, skills, and confidence acquired to effectively support implementation:
Thinking about addressing barriers and not simply providing MORE information and MORE education has stuck with me and has impacted how I approach other initiatives and projects.
Changing how I think about implementing or addressing any practice change. I am constantly thinking about what REAL barriers to change might be.
Impact on practice change
Data on the impact of the program on practice changes is available from the first cohort only, as funded projects in subsequent cohorts are still in progress. Eight teams were funded in the first cohort, but two were unable to complete their projects due to management changes and shifting priorities within their departments. Of the remaining six teams, three showed demonstrable practice changes across their respective practice areas, giving the program a 50% success rate:
-
1.
In a cardiac inpatient unit, depression screening increased from none to 75%, and follow-up discussions with primary care providers were recorded at 36%.
-
2.
In a tertiary hospital setting, malnutrition screening increased by 50%.
-
3.
In a hospital-based physical rehabilitation program, 67% of physical therapists and occupational therapists reported increased uptake of the Canadian Stroke Best Practice Recommendations.
Three teams completed their projects but were not able to demonstrate measurable practice change. Cohort 2 (n = 9 teams) and cohort 3 (n = 6 teams) are currently completing their projects, with promising results, including one team that has significantly increased the percentage of patients with spinal cord injury receiving functional electrical stimulation treatment. Another team has garnered strong interest in their topic—treating problematic methamphetamine use—and was approached by local health care policymakers as well as staff from the United States Office of National Drug Control Policy, wanting information on their implementation plan.
Reported challenges
Despite the positive impacts on practitioners’ KT capacity and demonstrable practice changes, many teams experienced challenges, common to most KT programs [5, 6, 13], and including:
-
Team member turnover
-
Manager turnover
-
Gaps in communication with mentor
-
Projects taking more time than anticipated
-
Lack of support from key stakeholders for the practice change
Steps were taken in response to these challenges, as outlined below.
Discussion
The evaluation of the KT Challenge program demonstrates that a multi-faceted implementation support program for HCPs is effective at moving evidence into practice and can be conducted within a modest budget ($5000 per team). The content of the KT Challenge workshops is designed to address the identified barriers to successful implementation of evidence into practice, including lack of knowledge, skills, time, and leaders’ endorsement [8,9,10]. The steady numbers of applicants for the KT Challenge program indicate sustained interest, including in the most recent cohort (October 2020), in the midst of the global pandemic, 13 teams submitting LOIs to join the program. This confirms the findings of the 2012 survey that showed 80% of health care respondents are interested in improving their knowledge and skills related to implementation [10].
Among challenges faced by teams, lack of time, changes in personnel, and shifting work priorities were the most commonly cited. The following steps were taken to improve the program for future cohorts:
-
Program leads provide feedback and suggestions to teams in response to their quarterly reports
-
Workshops and program documents were revised to specifically highlight the need for the project evaluation to collect data on the uptake of the practice changes
-
Program leads undertook more promotion of the program within their organizations to strengthen management’s commitment to the funded teams
However, even the teams who were unable to complete, or whose projects did not lead to measurable practice change, agreed that participation in the program was beneficial. We had hoped to see all projects result in demonstrable practice change arising from their interventions, but given the many barriers to practice improvements, it is unreasonable to expect 100% success. As Durlak and Dupre have noted, “Expecting perfect or near-perfect implementation is unrealistic. Positive results have often been obtained with levels around 60%...no study has documented 100% implementation for all providers.” (19, p. 331). In comparison, 50% of our completed teams reported evidence-based practice changes, demonstrating the effectiveness of the KT Challenge program. Furthermore, the 185 HCPs who participated in the program gained knowledge and confidence in KT practice skills, as well as improved ability to implement practice change.
We revised the KT Challenge program to enhance the relevance of the projects to patients and families and to improve the dissemination of the findings to audiences that include patients and families. Beginning with the 2019–2020 cohort, all KT Challenge teams were required to include a Patient-Family Partner on their implementation team, and all funding proposals are now reviewed by a panel that includes Patient-Family Partners.
Our evaluation shows that the KT Challenge has been very effective at increasing capacity in KT skills and has provided information on areas where the program can be improved. We have carefully reviewed evaluation comments from each cohort in the KT Challenge and have made adjustments to the program based on feedback from participants, offering progressively improved support with each cohort. We hope this support will lead to a higher percentage of teams demonstrating measurable practice changes.
These findings are based on the first cohort of teams who completed the entire program, which limited our ability to do more fulsome comparisons between the teams who were able to demonstrate practice improvements and those who were not able to complete their projects.
Conclusions
The KT Challenge program is a promising initiative that can be adopted across a variety of clinical settings to support the effective uptake of evidence-based practice. This evaluation demonstrates that clinicians will respond to opportunities for KT training, that enhanced capacity for KT skills is achievable with support, and—most importantly—that successful practice change can result from small-scale, mentored, and funded KT projects in clinical practice settings. This evaluation study contributes to the implementation science literature by providing a description of a new and effective model for embedding KT practice skills in health care settings.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Abbreviations
- HCP:
-
Health care professional
- KT:
-
Knowledge translation
- LOI:
-
Letter of Intent
- OT:
-
Occupational therapist
- PT:
-
Physiotherapist
References
National Health Service Sustainability Model. Downloaded 24 Sept 2020 from: https://webarchive.nationalarchives.gov.uk/20160805122935/http://www.nhsiq.nhs.uk/media/2757778/nhs_sustainability_model_-_february_2010_1_.pdf;2010. Accessed 1 Nov 2020. ISBN: 978-1-907045-36-3.
Eames S, Bennett S, Whitehead M, Fleming J, Shanling O, Mickan S, et al. A pre-post evaluation of a knowledge translation capacity-building intervention. Austr Occup Ther J. 2018;65(6):479–93.
Heaton J, Day J, Britten N. Inside the ‘black box’ of a knowledge translation program in applied health research. Qual Health Res. 2015;25(11):1477–91.
Park JS, Moore JE, Sayal R, Holmes BJ, Scarrow G, Graham ID, et al. Evaluation of the “Foundations in Knowledge Translation” training initiative: preparing end users to practice KT. Implement Sci. 2018;13(1):1748–5908.
Thomson D, Brooks S, Nuspl M, Hartling L. Programme theory development and formative evaluation of a provincial knowledge translation unit. Health Res Policy Syst. 2019;17(1):1478–4505 PMID: 30971263.
Bennett S, Whitehead M, Eames S, Fleming J, Low S, Caldwell E. Building capacity for knowledge translation in occupational therapy: learning through participatory action research. BMC Med Educ. 2016;16(1):1472–6920.
Goodenough B, Fleming R, Young M, Burns K, Jones C, Forbes F, et al. Gerontol Geriatr Educ. 2017;38(4):392–406.
Davis D, Evans M, Jadad A, Perrier L, Rath D, Ryan D, et al. The case for KT: shortening the journey from evidence to effect. BMJ. 2003;327:33e5.
Moore J, Rashid S, Park J, Khan S, Straus S. Longitudinal evaluation of a course to build core competencies in implementation practice. Implement Sci. 2018;13(1):106. https://doi.org/10.1186/s13012-018-0800-3.
The Michael Smith Foundation for Health Research. Training and resources to support research use: a BC needs assessment; 2012 downloaded 31 Aug 2020 from https://www.msfhr.org/sites/default/files/KT_Needs_Assessment_FINAL.pdf
Wenke R, Noble C, Weir KA, Mickan S. What influences allied health clinician participation in research in the public hospital setting: a qualitative theory-informed approach. BMJ Open. 2020;10(8):e036183. https://doi.org/10.1136/bmjopen-2019-036183.
Davis R, D’Lima D. Building capacity in dissemination and implementation science: a systematic review of the academic literature on teaching and training initiatives. Implement Sci. 2020;15(1):97. https://doi.org/10.1186/s13012-020-01051-6.
Gaglardi A, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci. 2016;11:38. https://doi.org/10.1186/s13012-016-0399-1.
Black A, Ali S, Baumbusch J, McNamee K, Mackay M. Practice-based nursing research: evaluation of clinical and professional impacts from a research training program. J Clin Nurs. 2019;1:7. https://doi.org/10.1111/jocn.14861.
Black A, Balneaves L, Garossino C, Puyat J, Qian H. Promoting evidence-based practice through a research training program for point-of-care clinicians. J Nurs Administr. 2015;45(1).
Geldhof GJ, Warner DA, Finders JK, Thogmartin AA, Clark A, Longway KA. Revisiting the utility of retrospective pre-post designs: the need for mixed-method pilot data. Eval Program Plan. 2018;70:83–9. https://doi.org/10.1016/j.evalprogplan.2018.05.002. Epub 2018 Jul 9. PMID: 30029016.
Acknowledgements
We would like to acknowledge the 185 HCPs who have taken part in the KT Challenge, the numerous Patient-Family Partners who have advised teams and reviewed proposals, and the leaders in our organizations who provide funding for the KT Challenge program.
Funding
The KT Challenge program is supported by Providence Health Care (PHC) Research Institute and the PHC Professional Practice Office, and Vancouver Coastal Health Research Institute and the Robert H. N. Ho Enhancing Patient Care Fund.
Author information
Authors and Affiliations
Contributions
AB and MS contributed to the overall KT Challenge program design and evaluation. AB and AC oversaw the program management and operations at their respective health organization sites. MS developed the workshop content, delivered the presentations, developed the evaluation, and consulted with the teams. KC, AH, JK, AL, MM, AS, and MES formed the program advisory committee to review project proposals and inform program decisions. All authors contributed to writing and revising the manuscript. The authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Black, A.T., Steinberg, M., Chisholm, A.E. et al. Building capacity for implementation—the KT Challenge. Implement Sci Commun 2, 84 (2021). https://doi.org/10.1186/s43058-021-00186-x
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s43058-021-00186-x