Skip to main content
Download PDF

A retrospective assessment of the KLIK PROM portal implementation using the Consolidated Framework for Implementation Research (CFIR)

Quality of Life Research volume 30, pages 3049–3061 (2021)Cite this article

Abstract

Purpose

The KLIK Patient-Reported Outcome Measure (PROM) portal is an evidence-based intervention implemented in clinical practice in > 25 Dutch hospitals for patients (children and adults) who regularly visit the outpatient clinic. Implementation science frameworks can be used to understand why implementation succeeded or failed, to structure barriers and enablers, and to develop implementation strategies to overcome barriers. This paper aimed to (A) retrospectively describe determinants of successful KLIK PROM implementation using the Consolidated Framework for Implementation Research (CFIR), and (B) identify current barriers and match implementation strategies.

Methods

(A) The KLIK implementation process was described retrospectively based on literature and experience, using the 39 CFIR constructs organized in five general domains: intervention characteristics, outer setting, inner setting, characteristics of individuals, and implementation process. (B) The CFIR-Expert Recommendations for Implementing Change (ERIC) Implementation Strategy Matching tool identified current barriers in the KLIK implementation and matched implementation strategies that addressed the identified barriers.

Results

(A) The most prominent determinants of successful KLIK PROM implementation lie in the following CFIR domains: intervention characteristics (e.g., easy to use), characteristics of individuals (e.g., motivation), and process of implementation (e.g., support). (B) 13 CFIR constructs were identified as current barriers for implementing the KLIK PROM portal. The highest overall advised ERIC strategy for the specific KLIK barriers was to identify and prepare champions.

Conclusion

Using an implementation science framework, e.g., CFIR, is recommended for groups starting to use PROMs in clinical care as it offers a structured approach and provides insight into possible enablers and barriers.

Introduction

Patient-Reported Outcome Measures (PROMs) are standardized, validated questionnaires that are completed by patients, such as a person’s perspective on their health, well-being, or symptoms [1, 2]. PROMs can be used for several purposes: at group level to study differences between disease populations, to describe the effects of treatment in clinical trials, and to assess quality of care or on an individual level to promote patient-centered care, guide clinical decision-making, and to facilitate communication [3]. There is widespread evidence for the effects of PROM applications on an individual level regarding an increase in Health-related Quality of Life (HRQOL) scores, satisfaction with care and communication about PROs in research settings, both in adult [4,5,6] and pediatric [7,8,9,10,11,12] samples. Yet the implementation of these evidence-based (EB) PROMs interventions is challenging.

The KLIK PROM portal (www.hetklikt.nu and www.klik-uk.org) is an example of an EB PROM intervention for patients (children or adults) who regularly visit the outpatient clinic [13]. Patients complete PROMs online, prior to their visit. Answers are transformed into an electronic PROfile (ePROfile; Fig. 1). Clinicians discuss this ePROfile with patients, to monitor well-being over time, identify problems, and provide tailored advice and interventions. The effects of using the KLIK PROM portal have been demonstrated in pediatric oncology [7] and in pediatric rheumatology [12], by showing an increased and more detailed discussion of HRQOL and psychosocial functioning during the consultation, less undetected problems, and a higher clinician-reported satisfaction with provided care, without lengthening the consultation duration.

Fig. 1
figure 1

a KLIK ePROfile—literal feedback of the individual items on the Pediatric Quality of Life Inventory (PedsQL) b KLIK ePROfile—graphical feedback of the PedsQL, including norm lines

Full size image

Despite the availability of several EB PROM interventions across the world, the actual implementation of PROM interventions in clinical practice remains limited [14,15,16,17]. There is a critical gap in behavioral medicine between what we know can optimize patient health and care outcomes and what gets implemented in everyday practice [1]. If EB PROM interventions are not successfully implemented in clinical practice, then intended effects are not reached, which limits the impact on patients’ health outcomes [18, 19]. Traditional randomized controlled trials study the effectiveness of PROM interventions under ideal circumstances. Yet for the implementation of PROMs in clinical practice, a different, more flexible approach is needed. Often, a “voltage drop” (a dramatic decrease in effectiveness) is seen once interventions get implemented in clinical practice [20]. Implementation research is defined by the National Institute of Health as the “scientific study of the use of strategies to adopt and integrate evidence-based health interventions into clinical and community settings in order to improve patient outcomes and benefit population health” [21]. Therefore, a scientific approach to the change process is crucial. In order to know what drives successful implementation of PROMs in clinical practice, we need to study the mechanisms that influence implementation outcomes [17, 22, 23]. Implementation science models, theories, or frameworks support in identifying factors that influence an implementation process or outcome.

In general, three overarching aims of theoretical approaches and five categories of theories, models, and frameworks used in implementation science can be distinguished [24]: (1) guiding the process of translating research into practice (process models), (2) understanding and/or explaining what influences implementation outcomes (determinant frameworks, classic theories, and implementation theories), and (3) evaluating implementation (evaluation frameworks). Specifically, determinant frameworks are useful in understanding or explaining what influences implementation outcomes and to support the design of implementation strategies or maximizing the use of enablers to implementation [24].

A widely cited and comprehensive determinant framework in the implementation science literature in health is the Consolidated Framework for Implementation Research (CFIR). Damschroder et al. [22] aimed to develop a framework that comprises common constructs from published implementation theories and includes, therefore, missing key constructs in other theories. It contains 39 constructs which are organized in five general domains: (1) intervention characteristics (e.g., evidence, complexity, adaptability, costs), (2) outer setting (e.g., peer pressure and external policies), (3) inner setting (e.g., structural characteristics, implementation climate, and culture), (4) characteristics of individuals (e.g., knowledge about the intervention and self-efficacy), and (5) implementation process (e.g., planning, engaging stakeholders, champions, and execution), see Fig. 3. Determinant frameworks, such as CFIR, are specifically useful in understanding or explaining what influences implementation outcomes and to support the design of implementation strategies or maximizing the use of enablers to implementation [24]. This paper aimed to (A) retrospectively describe the most prominent determinants and reasons of successful KLIK PROM implementation using CFIR and (B) use the CFIR-ERIC Implementation Strategy Matching tool to identify current barriers of the KLIK PROM portal implementation and match implementation strategies that address the identified barriers. In our specific study context, the CFIR framework seemed particularly useful as it covers a wide range of implementation constructs and domains and it allowed us to use a standardized framework to explain the influence of each domain on the implementation outcomes of an evidence-based PROM portal. With years of experience in the development and implementation, the KLIK PROM portal is now in a phase of understanding what barriers and facilitators have already been resolved and determining what major determinants are currently of influence to move to the next area of implementation: sustainability.

Methods

The evidence-based KLIK PROM portal

The development and implementation of the KLIK PROM portal is based on multiple studies (Supplemental Table 1). The predecessor of the KLIK ePROfile was the QLIC-ON PROfile [25]. During the QLIC-ON study, two generic HRQOL questionnaires widely used in pediatrics (TAPQOL [26] and PedsQL [27]) were converted into digital questionnaires. Patients were asked to complete a HRQOL questionnaire on a laptop in the waiting room of the outpatient clinic, prior to the visit. The literal answers and graphs were printed out, fed back to the pediatrician in a QLIC-ON PROfile on paper, and discussed with patients and parents during the consultation [25]. However, completing PROMs at the outpatient clinic and providing hard copy PROfiles was logistically complicated, and therefore, they are hard to implement in a real-world situation. As a result, the KLIK website (www.hetklikt.nu) was developed during the KLIK study in pediatric rheumatology [28]. From that moment, children and parents completed the questionnaires online at home. The implementation of KLIK, as part of standard care, started in 2011 [7, 12]. To gain more insight into the implementation process and outcomes, a study was conducted to identify barriers and enablers in this process in pediatric oncology [29].

Table 1 Description of the most prominent determinants of successful KLIK implementation using CFIR
Full size table

Currently, KLIK is part of standard care in > 70 different patient groups (e.g., diabetes, nephrology) in > 20 centers in the Netherlands and 3 centers in the United Kingdom. Over 17,000 patients are registered on the KLIK website and around 1,000 clinicians (e.g., physicians, nurses, psychologists) have been trained in the use of KLIK. KLIK is implemented in various settings, including hospital outpatient clinics, rehabilitation centers, and recently in dentistry. KLIK was initially developed for use in pediatrics, but since 2017 KLIK has also been implemented in adult care (e.g., coagulation diseases and medical psychology). The KLIK expert team of the Emma Children’s Hospital Amsterdam UMC coordinates the implementation of the KLIK PROM portal in pediatrics and adult healthcare in 20 hospitals in the Netherlands. The KLIK expert team in the Princess Máxima Center for pediatric oncology coordinates the implementation of KLIK in this center. KLIK can be implemented for any patient group, on request of a multidisciplinary team. The implementation procedure of the KLIK PROM portal has previously been described according to the guidelines of the International Society for Quality of Life Research (ISOQOL) [13]. A core element of the KLIK implementation process is to train all team members in the use of KLIK and discussing PROMs in the consultation room. A summary of the implementation process is shown in Fig. 2.

Fig. 2
figure 2

Overview of the KLIK implementation process for one multidisciplinary team. * The KLIK expert team consists of researchers with expertise in the field of (implementing) PROMs and HRQOL research

Full size image

Design

To retrospectively assess the KLIK PROM implementation using the CFIR framework, a mixed methods design was used. Part A consisted of a qualitative description regarding the most prominent determinants of successful KLIK PROM implementation. Part B consisted of an evaluation of current barriers in the KLIK implementation process and matching potential future strategies to reduce these barriers using the CFIR-ERIC Implementation Strategy Matching tool v1.0 [30, 31] and a qualitative description of the identified barriers and strategies that have been used already by the KLIK expert team.

A. Retrospectively describing the most prominent determinants of successful KLIK PROM implementation using CFIR

The CFIR framework was used to retrospectively describe the implementation process of the KLIK PROM portal in different patient groups and hospitals throughout the Netherlands and to identify determinants in this process. Only the determinants relevant for the KLIK implementation process were described. To define which determinants were relevant for successful KLIK PROM implementation the following steps were taken. First, the KLIK PROM implementation process was described and discussed by the KLIK expert team, using all 39 CFIR constructs. However, for the reason of clarity, only the most prominent CFIR determinants relevant for the KLIK PROM implementation were extracted here (see Table 1). Second, the authors discussed which facilitators they found most prominent to describe. If the majority of authors considered a CFIR construct as valuable, it was included in the qualitative description. The KLIK implementation process was described based on published literature regarding the development, effectiveness, and implementation of KLIK in various settings and options for visual feedback of the PROMs (Supplemental Table 1) and unpublished literature (e.g., the KLIK user manual and training) about the KLIK portal and on experiences of the KLIK expert team.

B. CFIR-ERIC Implementation Strategy Matching Tool to identify current barriers of the KLIK PROM portal implementation

The CFIR-ERIC Implementation Strategy Matching tool v1.0 [31] was used to identify current barriers in the ongoing KLIK implementation and to match implementation strategies that address the identified barriers. The CFIR-ERIC tool is based on the CFIR framework and the 73 Expert Recommendations for Implementing Change (ERIC) implementation strategies [32]. During the development of this tool [30], implementation researchers and clinicians (panelists) were presented with brief descriptions of barriers based on CFIR construct definitions. They were asked to rank implementation strategies that would best address each barrier.

Within the provided Excel tool, one can indicate which CFIR constructs are barriers to implementation. Five KLIK expert team members based in the Emma Children’s hospital Amsterdam UMC and three in the Princess Máxima Center for pediatric oncology involved in the implementation of the KLIK PROM portal independently indicated which of the 39 CFIR constructs were perceived as current barriers in the overall KLIK implementation. These eight expert team members include all authors. When the majority (5 or more members) of the KLIK expert team identified a CFIR construct as barrier, this was entered in the matching tool. Specific agreement (both positive and negative, including 95% confidence intervals) was calculated according to De Vet et al. [33] using R.

Consequently, the tool provided output with percentages showing which ERIC implementation strategies can best be used to reduce these specific CFIR barriers. Percentages reflect the proportion of panelists endorsing a strategy appropriate for that barrier. Strategies are sorted by the cumulative percentage value. According to the tool, the strategies with the highest cumulative percentages are most effective in reducing the combined identified barriers [30]. In the results, the ten highest cumulative percentages, and, therefore, the overall advised strategies for the specific KLIK barriers will be shown. In addition, for every identified barrier using the CFIR-ERIC tool, the authors discussed what was already done in the past to reduce the impact of this barrier on the KLIK implementation process and the reasons why it still remains a barrier.

Results

A. Retrospectively describing the most prominent determinants of successful KLIK PROM implementation using CFIR

Based on previous research and on multiple years of experience implementing the KLIK PROM portal in clinical practice, the most prominent determinants were identified by the KLIK expert team (Fig. 3) and reasons for successful KLIK implementation are depicted in Table 1.

Fig. 3
figure 3

Overview of the five domains of CFIR, indicating determinants and barriers for the implementation of the KLIK PROM portal. Most prominent determinants are indicated in italics. The 13 identified current barriers using the CFIR-ERIC matching tool are indicated in bold

Full size image

Several of the CFIR constructs were not applicable to the implementation of the KLIK PROM portal, unknown or differ too much between the different multidisciplinary teams and hospitals. These constructs include patient needs, networks & communications, culture, relative priority, learning climate, individual identification with organization, other personal attributes, and executing.

B. CFIR-ERIC Implementation Strategy Matching Tool to identify current barriers of the KLIK PROM portal implementation

Of the 39 CFIR constructs, 13 were identified by the KLIK expert team as current barriers for implementing the PROM portal using the CFIR-ERIC matching tool. The total specific agreement was 68.1% (95% CI 59.6%–77.6%), positive agreement (CFIR barrier) was 75.9% (95% CI 68.1%–84.6%), and negative agreement (no CFIR barrier) was 53.1% (95% CI 44.0%–63.2%). In Table 2 and Fig. 3, the 13 barriers are shown. Per barrier is described what is already done as well as the challenges that remain.

Table 2 Identified current barriers for the KLIK PROM portal implementation using the CFIR-ERIC matching tool
Full size table

Matching ERIC strategies to CFIR barriers

The identified barriers were matched to the 73 ERIC strategies using the CFIR-ERIC matching tool. Of these ERIC implementation strategies, the top 10 strategies matching the 13 identified CFIR barriers are shown in Table 3, sorted by the cumulative percentage value. Percentages reflect the proportion of panelists endorsing a strategy for that specific CFIR barrier. The tool shows that the strategy ‘identify and prepare champions’ is most effective in addressing the combination of identified barriers, followed by ‘promote adaptability’ and ‘assess for readiness and identify barriers and facilitators’.

Table 3 Output of the CFIR-ERIC matching tool: top 10 ERIC strategies matched to the 13 identified CFIR barriers for current KLIK implementation
Full size table

Discussion

This paper aimed to retrospectively describe the most prominent determinants of successful KLIK PROM portal implementation using the Consolidated Framework for Implementation Research (CFIR) and to identify current barriers and matching implementation strategies for the KLIK implementation using the CFIR-ERIC Implementation Strategy Matching Tool.

This retrospective evaluation shows that the strength of the KLIK PROM portal implementation lies particularly in the following CFIR domains: intervention characteristics (e.g., easy to use, direct feedback), characteristics of individuals (e.g., motivated clinicians), and process of implementation (e.g., support of the KLIK expert team). In addition, the climate of the outer setting is changing and patient-reported outcomes are more valued, which facilitates the implementation of the KLIK PROM portal. On the other hand, barriers in the implementation lie mainly in the domain of the inner setting and the intervention characteristics. Regarding the inner setting, involving and motivating all stakeholders at various levels of the multidisciplinary teams and hospitals is challenging. Regarding the intervention characteristics, mainly the tension field of providing optimal support of the KLIK expert team and the use of the KLIK PROM portal on the one hand and keeping low costs on the other hand is difficult. These findings are in line with another study discussing PROM implementation [38], where the authors describe the same relevant CFIR domains. This implies that the CFIR domain ‘outer setting’ might be less relevant than the other four domains when describing PROM implementation. However, a recent study on PREM implementation did find relevant outcomes regarding the outer setting, or macro level [39], and other literature on PROMs in palliative care also conclude that all CFIR domains need consideration for effective implementation [40].

Most CFIR domains were applicable to implementation of the KLIK PROM portal, showing that CFIR can be used in the context of implementing PROMs. However, the framework is not specifically developed for this context, resulting in insufficient attention for specific parts of the PROM implementation. For example, the content, length, and psychometric properties of PROMs are important factors for successful implementation of PROMs in clinical practice and are not addressed by the CFIR framework.

The CFIR is a comprehensive framework based on various published implementation theories [22], resulting in a very extensive framework consisting of many constructs, which can make it complicated to use. The five domains of the framework are intertwined and interacting, making it hard to determine where points of attention can be placed without iteration. In particular, the domain inner setting consists of many overlapping subdomains with intangible concepts. In addition, a recent systematic review on implementing e-health interventions shows blind spots in current literature about contextual factors (such as the organization), which makes it difficult for clinicians and researchers to understand these concepts and to translate it to clinical practice [41]. In previous literature, other weaknesses of CFIR are mentioned. In their systematic review on PROM implementation, Foster et al. identified the importance of different stages of the implementation process, which is not captured by CFIR [1].

The CFIR can be described as a determinant framework [24]. Determinant frameworks specify which factors (determinants) have a facilitating or inhibiting effect on the implementation. These frameworks thus describe the influence of processes on the implementation outcomes, but do not address these implementation outcomes, in contrary to evaluation frameworks. Therefore, it would be useful to use the CFIR in combination with another type of model. For instance, a widely used model on implementation outcomes is the “conceptual model of implementation research”, as described by Proctor and colleagues [18]. In order to improve outcomes for patients, it is important to be able to determine which determinants relate to which specific implementation outcomes. Only then can be reliably concluded which specific strategies work for which implementation outcomes.

The CFIR-ERIC Implementation Strategy Matching tool provided implementation strategies for the identified CFIR barriers [30]. Some of the suggested implementation strategies can be explored and used in the KLIK PROM portal implementation in the upcoming years. For example, assess key stakeholders for readiness is an ongoing process and still a challenge. By conducting individual interviews with the more reluctant clinicians underlying resistance can be better understand and addressed. In addition, identifying expected barriers and facilitators in the implementation process by actively discussing these topics in multidisciplinary team meetings in a more structured way is necessary. Also, incentives for patients in using the KLIK PROM portal could be explored further by increasing patient engagement.

However, not all suggested strategies by the matching tool provided new insights as they were directly linked to the perceived barrier (e.g., identify and prepare champions for the barrier ‘champions’ and access new funding for the barrier ‘cost’) and therefore were already known by the KLIK expert team. In addition, some strategies are currently being worked on (e.g., tailoring strategies, inform local opinion leaders, and identify barriers in the implementation process). Though, these strategies are difficult to implement and the tool underlines the need to pay more attention to these important strategies.

To further improve the KLIK implementation process in daily clinical practice, both the identified current barriers as well as the strategies extracted from the CFIR-ERIC tool can be used, to provide some examples:

  • Recently, more and more evidence has become available for the relative advantage of implementing PROMs [42, 43]. We incorporate this information in the training to clinicians (step 4 in Fig. 2) and in the information we send to interested stakeholders to overcome this barrier. This might also affect the barrier tension for change.

  • To overcome the barrier of structural characteristics, creating awareness within the board of hospitals to facilitate larger scale implementation can be an opportunity. This might also affect the barrier leadership engagement.

  • Regarding engaging key stakeholders, patients and patient associations should be more involved in e.g., selecting PROs and PROMs and choices regarding frequency (step 1 in Fig. 2).

On the other hand, some current barriers will likely remain or even become more prominent in the future. For example complexity, due to increased privacy legislation, the KLIK PROM portal requires now the use of two-factor authentication, which does not benefit the usability of KLIK for some users.

At the time the implementation of the KLIK PROM portal in clinical practice started, a variety of implementation frameworks (including CFIR) and instruments to monitor and evaluate the implementation process from the start were not yet available. Just as we have evolved as a group, implementation science has evolved over the past decade as well. Implementation of the KLIK PROM portal was therefore essentially a process of “learning by doing”. Each time a specific multidisciplinary team showed interest in using KLIK, novel challenges appeared. As a result, a wide range of implementation strategies were used to tackle these particular issues. Notably, without realizing it at the time, many of the principles and strategies that are outlined in the CFIR tool were applied.

We recommend groups starting to implement PROMs in their setting to use an implementation science framework, like the CFIR, as knowing which factors need to be taken into account can lead to a more successful implementation in a specific context. The CFIR authors have developed an Interview Guide Tool (https://cfirguide.org/tools/) that can help researchers to question constructs of the CFIR that apply for the specific context. As every individual implementation process is different, also the constructs that are applicable differ.

Strengths of this study include the broad view of the retrospective description; multiple populations and multicenter experiences have been taken into account. In addition, the description is based on long-term experience and on published literature. However, this paper has several limitations. First, although a deliberate choice, no standardized qualitative research methods were used in this paper as the aim of this paper was to give a retrospective description of the KLIK PROM implementation process using the CFIR framework with the overarching purpose to create more awareness for the use of implementation science in PROM research. Second, the determinants and barriers for successful KLIK PROM implementation were described based on the experiences of the KLIK expert team (existing of members from two different centers) and this could have led to a selective view from the KLIK expert team. However, the KLIK expert team works closely with a variety of stakeholders on a day-to-day basis, including clinicians, patients, and parents. They furthermore provide opportunities for stakeholders to provide feedback during regular evaluation meetings. In addition, recently two evaluation studies were carried out to gain more insight into the perspectives of clinicians [35], and pediatric patients and parents [44]. Thus, even though other stakeholders were not literally represented as co-authors, it can be assumed that their opinions are represented throughout this study.

In conclusion, this retrospective approach showed that the CFIR provides clinicians and scientists guidance during a healthcare implementation process and can be used in all phases of implementation, although it is a quite extensive and complex framework with some overlapping constructs. For example, the CFIR can be used retrospectively, reflected in this article, to describe the implementation process and its determinants and to identify remaining barriers. An advantage of using this theoretical framework prior to start of implementation is that clinicians become aware of the possible facilitating determinants and barriers for implementation. Using an implementation science framework, like the CFIR, is recommended for groups starting to use PROMs in clinical care as knowing which factors need to be taken into account can lead to a more successful implementation in a specific context.

References

  1. Foster, A., Croot, L., Brazier, J., Harris, J., & O'Cathain, A. (2018). The facilitators and barriers to implementing patient reported outcome measures in organisations delivering health related services: A systematic review of reviews. Journal of Patient Reported Outcomes, 2, 46. https://doi.org/10.1186/s41687-018-0072-3.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Connolly, M. A., & Johnson, J. A. (1999). Measuring quality of life in paediatric patients. Pharmacoeconomics, 16(6), 605–625.

    CAS  Article  Google Scholar 

  3. Greenhalgh, J. (2009). The applications of PROs in clinical practice: What are they, do they work, and why? Quality of Life Research, 18(1), 115–123. https://doi.org/10.1007/s11136-008-9430-6.

    Article  PubMed  Google Scholar 

  4. Kotronoulas, G., Kearney, N., Maguire, R., Harrow, A., Di Domenico, D., Croy, S., et al. (2014). What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials. Journal of Clinical Oncology, 32(14), 1480–1501. https://doi.org/10.1200/jco.2013.53.5948.

    Article  PubMed  Google Scholar 

  5. Velikova, G., Booth, L., Smith, A. B., Brown, P. M., Lynch, P., Brown, J. M., et al. (2004). Measuring quality of life in routine oncology practice improves communication and patient well-being: A randomized controlled trial. Journal of Clinical Oncology, 22(4), 714–724.

    Article  Google Scholar 

  6. Detmar, S. B., Muller, M. J., Schornagel, J. H., Wever, L. D., & Aaronson, N. K. (2002). Health-related quality-of-life assessments and patient-physician communication: A randomized controlled trial. JAMA, 288(23), 3027–3034.

    Article  Google Scholar 

  7. Engelen, V., Detmar, S., Koopman, H., Maurice-Stam, H., Caron, H., Hoogerbrugge, P., et al. (2012). Reporting health-related quality of life scores to physicians during routine follow-up visits of pediatric oncology patients: Is it effective? Pediatric Blood & Cancer, 58(5), 766–774. https://doi.org/10.1002/pbc.23158.

    Article  Google Scholar 

  8. de Wit, M., Delemarre-van de Waal, H. A., Bokma, J. A., Haasnoot, K., Houdijk, M. C., Gemke, R. J., et al. (2010). Follow-up results on monitoring and discussing health-related quality of life in adolescent diabetes care: Benefits do not sustain in routine practice. Pediatr Diabetes, 11(3), 175–181. https://doi.org/10.1111/j.1399-5448.2009.00543.x.

    Article  PubMed  Google Scholar 

  9. de Wit, M., Delemarre-van de Waal, H. A., Bokma, J. A., Haasnoot, K., Houdijk, M. C., Gemke, R. J., et al. (2008). Monitoring and discussing health-related quality of life in adolescents with type 1 diabetes improve psychosocial well-being: A randomized controlled trial. Diabetes Care, 31(8), 1521–1526. https://doi.org/10.2337/dc08-0394.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Wolfe, J., Orellana, L., Cook, E. F., Ullrich, C., Kang, T., Geyer, J. R., et al. (2014). Improving the care of children with advanced cancer by using an electronic patient-reported feedback intervention: Results from the PediQUEST randomized controlled trial. Journal of Clinical Oncology, 32(11), 1119–1126. https://doi.org/10.1200/jco.2013.51.5981.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Wolfe, J., Orellana, L., Ullrich, C., Cook, E. F., Kang, T. I., Rosenberg, A., et al. (2015). Symptoms and distress in children with advanced cancer: Prospective patient-reported outcomes from the PediQUEST study. Journal of Clinical Oncology, 33(17), 1928–1935. https://doi.org/10.1200/jco.2014.59.1222.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Haverman, L., Van Rossum, M. A., Van Veenendaal, M., van den Berg, J. M., Dolman, K. M., Swart, J., et al. (2013). Effectiveness of a web-based application to monitor health-related quality of life. Pediatrics, 131(2), 533–543. https://doi.org/10.1542/peds.2012-0958.

    Article  Google Scholar 

  13. Haverman, L., van Oers, H. A., Limperg, P. F., Hijmans, C. T., Schepers, S. A., Sint Nicolaas, S. M., et al. (2014). Implementation of electronic patient reported outcomes in pediatric daily clinical practice: The KLIK experience. Clinical Practice in Pediatric Psychology, 2(1), 50–67.

    Article  Google Scholar 

  14. Jensen, R. E., Snyder, C. F., Abernethy, A. P., Basch, E., Potosky, A. L., Roberts, A. C., et al. (2014). Review of electronic patient-reported outcomes systems used in cancer clinical care. J Oncol Pract, 10(4), e215–222. https://doi.org/10.1200/jop.2013.001067.

    Article  PubMed  Google Scholar 

  15. Hasson, H. (2010). Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implementation Science, 5, 67. https://doi.org/10.1186/1748-5908-5-67.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Sung, L. (2015). Priorities for quality care in pediatric oncology supportive care. Journal of Oncological Practice, 11(3), 187–189. https://doi.org/10.1200/jop.2014.002840.

    Article  Google Scholar 

  17. Leahy, A. B., Feudtner, C., & Basch, E. (2017). Symptom monitoring in pediatric oncology using patient-reported outcomes: Why, how, and where next. Patient. https://doi.org/10.1007/s40271-017-0279-z.

    Article  Google Scholar 

  18. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., et al. (2011). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7.

    Article  Google Scholar 

  19. Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009). Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health, 36(1), 24–34. https://doi.org/10.1007/s10488-008-0197-4.

    Article  PubMed  Google Scholar 

  20. Rothwell, P. M. (2005). External validity of randomised controlled trials: "To whom do the results of this trial apply?". Lancet, 365(9453), 82–93. https://doi.org/10.1016/s0140-6736(04)17670-8.

    Article  PubMed  Google Scholar 

  21. https://grants.nih.gov/grants/guide/pa-files/PAR-18-007.html (2019).

  22. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, 50. https://doi.org/10.1186/1748-5908-4-50.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Fleuren, M., Wiefferink, K., & Paulussen, T. (2004). Determinants of innovation within health care organizations: Literature review and Delphi study. International Journal for Quality in Health Care, 16(2), 107–123. https://doi.org/10.1093/intqhc/mzh030.

    Article  PubMed  Google Scholar 

  24. Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation Science, 10, 53. https://doi.org/10.1186/s13012-015-0242-0.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Engelen, V., Haverman, L., Koopman, H., et al. (2010). Development and implementation of a patient reported outcome intervention (QLIC-ON PROfile) in clinical paediatric oncology practice. Patient Education and Counseling, 81(2), 235–244. https://doi.org/10.1016/j.pec.2010.02.003.

    Article  PubMed  Google Scholar 

  26. Fekkes, M., Theunissen, N. C., Brugman, E., Veen, S., Verrips, E. G., Koopman, H. M., et al. (2000). Development and psychometric evaluation of the TAPQOL: A health-related quality of life instrument for 1-5-year-old children. Quality of Life Research, 9(8), 961–972.

    CAS  Article  Google Scholar 

  27. Varni, J. W., Burwinkle, T. M., Seid, M., & Skarr, D. (2003). The PedsQL 4.0 as a pediatric population health measure: Feasibility, reliability, and validity. Ambulatory Pediatrics, 3(6), 329–341.

    Article  Google Scholar 

  28. Haverman, L., Engelen, V., Van Rossum, M. A., Heymans, H. S., & Grootenhuis, M. A. (2011). Monitoring health-related quality of life in paediatric practice: Development of an innovative web-based application. BMC Pediatrics, 11, 3–10. https://doi.org/10.1186/1471-2431-11-3.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Schepers, S. A., Sint Nicolaas, S. M., Haverman, L., Wensing, M., Schouten van Meeteren, A. Y., Veening, M. A., et al. (2017). Real-world implementation of electronic patient-reported outcomes in outpatient pediatric cancer care. Psycho-oncology, 26(7), 951–959.

    Article  Google Scholar 

  30. Waltz, T. J., Powell, B. J., Fernández, M. E., Abadie, B., & Damschroder, L. J. (2019). Choosing implementation strategies to address contextual barriers: Diversity in recommendations and future directions. Implementation Science, 14(1), 42. https://doi.org/10.1186/s13012-019-0892-4.

    Article  PubMed  PubMed Central  Google Scholar 

  31. CFIR-ERIC Matching Tool v1.0. (2019). https://cfirguide.org/choosing-strategies/.

  32. Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., et al. (2015). A refined compilation of implementation strategies: Results from the expert recommendations for implementing change (ERIC) project. Implementation Science, 10(1), 21. https://doi.org/10.1186/s13012-015-0209-1.

    Article  PubMed  PubMed Central  Google Scholar 

  33. de Vet, H. C. W., Dikmans, R. E., & Eekhout, I. (2017). Specific agreement on dichotomous outcomes can be calculated for more than two raters. Journal of Clinical Epidemiology, 83, 85–89. https://doi.org/10.1016/j.jclinepi.2016.12.007.

    Article  PubMed  Google Scholar 

  34. Santana, M. J., Haverman, L., Absolom, K., Takeuchi, E., Feeny, D., Grootenhuis, M., et al. (2015). Training clinicians in how to use patient-reported outcome measures in routine clinical practice. Quality of Life Research, 24(7), 1707–1718.

    Article  Google Scholar 

  35. Teela, L., Van Muilekom, M. M., Kooij, L. H., Gathier, A. W., Van Goudoever, J. B., Grootenhuis, M. A., et al. (2020). Clinicians’ perspective on the implemented KLIK PROM portal in clinical practice. Quality of Life Research. https://doi.org/10.1007/s11136-020-02522-5.

    Article  PubMed  Google Scholar 

  36. Haverman, L., van Oers, H. A., van Muilekom, M. M., & Grootenhuis, M. A. (2019). Options for the interpretation of and recommendations for acting on different PROMs in daily clinical practice using KLIK. Medicine Care, 57, 52–58. https://doi.org/10.1097/MLR.0000000000001061.

    Article  Google Scholar 

  37. Snyder, C. F., Smith, K. C., Bantug, E. T., Tolbert, E. E., Blackford, A. L., & Brundage, M. D. (2017). What do these scores mean? Presenting patient-reported outcomes data to patients and clinicians to improve interpretability. Cancer, 123(10), 1848–1859. https://doi.org/10.1002/cncr.30530.

    Article  PubMed  Google Scholar 

  38. van Cranenburgh, O. D., Ter Stege, J. A., de Korte, J., de Rie, M. A., Sprangers, M. A., & Smets, E. M. (2016). Patient-reported outcome measurement in clinical dermatological practice: Relevance and feasibility of a web-based portal. Dermatology, 232(1), 64–70. https://doi.org/10.1159/000440613.

    Article  PubMed  Google Scholar 

  39. van Rooijen, M., Lenzen, S., Dalemans, R., Moser, A., & Beurskens, A. (2020). Implementation of a patient reported experience measure in a Dutch disability care organisation: A qualitative study. Journal of Patient Reporting Outcomes, 4(1), 5. https://doi.org/10.1186/s41687-019-0169-3.

    Article  Google Scholar 

  40. Pinto, C., Bristowe, K., Witt, J., Davies, J. M., de Wolf-Linder, S., Dawkins, M., et al. (2018). Perspectives of patients, family caregivers and health professionals on the use of outcome measures in palliative care and lessons for implementation: A multi-method qualitative study. Annals of Palliative Medicine, 7(Suppl 3), S137–s150. https://doi.org/10.21037/apm.2018.09.02.

    Article  PubMed  Google Scholar 

  41. Christie, H. L., Bartels, S. L., Boots, L. M. M., Tange, H. J., Verhey, F. J. J., & de Vugt, M. E. (2018). A systematic review on the implementation of eHealth interventions for informal caregivers of people with dementia. Internet Intervention, 13, 51–59. https://doi.org/10.1016/j.invent.2018.07.002.

    Article  Google Scholar 

  42. Basch, E., Deal, A. M., Dueck, A. C., Scher, H. I., Kris, M. G., Hudis, C., et al. (2017). Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA, 318(2), 197–198. https://doi.org/10.1001/jama.2017.7156.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Denis, F., Basch, E., Septans, A. L., Bennouna, J., Urban, T., Dueck, A. C., et al. (2019). Two-year survival comparing web-based symptom monitoring vs routine surveillance following treatment for lung cancer. JAMA, 321(3), 306–307. https://doi.org/10.1001/jama.2018.18085.

    Article  PubMed  PubMed Central  Google Scholar 

  44. van Muilekom, M. M. & Teela, L., van Oers, H. A., Grootenhuis, M. A., & Haverman, L. (In press). Patients’ and parents’ perspective on the implemented KLIK PROM portal in clinical practice.

Download references

Acknowledgements

This paper was reviewed and endorsed by the International Society for Quality of Life Research (ISOQOL) Board of Directors as an ISOQOL publication and does not reflect an endorsement of the ISOQOL membership. We thank all patients, caregivers, clinicians, and everyone else involved in the implementation of the KLIK PROM portal. In addition, we thank Biomedia for the continuous development of the KLIK website, Maud van Muilekom, Anouk Gathier and Layla Teunissen for rating the CFIR-ERIC matching tool, and Michiel Luijten for his methodological support.

Author information

Author notes

  1. Hedy A. van Oers and Lorynn Teela shared first.

Affiliations

  1. Psychosocial Department, G8-136, Emma Children’s Hospital/Amsterdam UMC, University of Amsterdam, Meibergdreef 9, Postbox 2266, 1100 DD, Amsterdam, The Netherlands

    Hedy A. van Oers, Lorynn Teela & Lotte Haverman

  2. Princess Máxima Center for Pediatric Oncology, Utrecht, The Netherlands

    Sasja A. Schepers & Martha A. Grootenhuis

Authors
  1. Hedy A. van Oers
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lorynn Teela
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sasja A. Schepers
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Martha A. Grootenhuis
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lotte Haverman
    View author publications

    You can also search for this author in PubMed Google Scholar

Consortia

the ISOQOL PROMs and PREMs in Clinical Practice Implementation Science Group

Corresponding author

Correspondence to Hedy A. van Oers.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 25 kb)

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

van Oers, H.A., Teela, L., Schepers, S.A. et al. A retrospective assessment of the KLIK PROM portal implementation using the Consolidated Framework for Implementation Research (CFIR). Qual Life Res 30, 3049–3061 (2021). https://doi.org/10.1007/s11136-020-02586-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11136-020-02586-3

Keywords

  • Implementation science
  • PROMs
  • Clinical practice
  • Framework