Abstract
Purpose
Since 2011, the evidence-based KLIK Patient Reported Outcome Measure (PROM) portal has been implemented in clinical practice in > 20 Dutch hospitals. Patients and/or parents complete PROMs on Health Related Quality of Life, symptoms and psychosocial functioning before their outpatient consultation. Answers are converted into an ePROfile and discussed by clinicians during consultation to monitor well-being over time and detect problems early. This study aims to get insight into the KLIK implementation from the clinician’s perspective.
Methods
As part of the KLIK implementation process, annual meetings were held with multidisciplinary teams to evaluate the use of KLIK. An online questionnaire was sent regarding (1) overall satisfaction, (2) feeling competent to discuss PROMs, (3) use of KLIK during the consultation, (4) influence of KLIK on the consultation, (5) usability of the KLIK PROM portal, (6) satisfaction with PROMs and feedback, and (7) support of the KLIK expert team. Open questions about (dis)advantages were included. Descriptive analyses were used.
Results
One hundred and forty-eight clinicians (response-rate 61%) from 14 hospitals in the Netherlands participated. Results show that: (1) clinicians report an overall satisfaction of median = 69/100 (visual analogue scale), (2) 85.8% feel competent discussing the ePROfile, (3) 70.3% (almost) always discuss the ePROfile, (4) 70.3% think that KLIK improves consultation, (5) 71.6% think KLIK is easy to use, (6) 80.4% are satisfied with the feedback of the overall KLIK ePROfile, (7) 71.6% experience sufficient support of the KLIK team.
Conclusion
Participating clinicians are generally satisfied with KLIK. Improvements to the KLIK PROM portal are now realized based on the mentioned disadvantages (e.g., shorten PROM completion by use of PROMIS and integrating KLIK with Electronic Health Records).
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
In the past decades, there has been increased attention for the use of Patient Reported Outcome Measures (PROMs) in daily clinical practice enabling patient-centered care [1]. Discussing PROMs in the consultation room empowers patients, enhances patient-clinician communication, and promotes shared decision making [2,3,4,5]. Monitoring patients by using PROMs increases awareness for patients’ concerns, facilitates recognition of physical or psychological problems, improves patient satisfaction with health care, and is associated with improved treatment outcomes, including survival [3, 4, 6,7,8].
After two efficacy studies [9, 10], the KLIK PROM portal (www.hetklikt.nu) is being implemented in daily clinical practice since 2011. These studies showed that the feedback and discussion of PROMs in the consultation room resulted in more attention for, and improved identification of, psychosocial and emotional problems and increased satisfaction of pediatricians with the provided care [9, 10]. Within the KLIK PROM portal, pediatric patients (≥ 8 years) and/or their parents and adult patients are asked to complete PROMs regarding Health Related Quality of Life (HRQOL), symptoms and/or psychosocial functioning online at home prior to the outpatient consultation. The answers are converted into an electronic PROfile (KLIK ePROfile, Fig. 1) that contains a broad range of feedback options tailored to each specific PROM [11]. The clinician discusses the KLIK ePROfile during the outpatient consultation with patients and/or parents in order to monitor well-being over time, detect problems at an early stage and provide tailored advice and interventions. Currently, more than 17,000 patients from 70 different patient groups (e.g., rheumatology, diabetes, oncology) have registered themselves on the KLIK website and around 1,000 clinicians (e.g., physicians, nurses, psychologists, social workers, physiotherapists, dieticians, and speech therapists) have been trained (around 800 active users) in the use of KLIK in daily clinical practice in > 20 different hospitals in the Netherlands [12] and 3 hospitals in the United Kingdom (www.klik-uk.org).
Nevertheless, implementing a PROM portal in clinical practice is a challenging process in which the interests of different stakeholders are involved [12, 13]. For a successful implementation, different determinants can be distinguished on the level of intervention characteristics, the clinician, the patient (and parent), and the socio-political context. In the past years, the intervention characteristics of the KLIK PROM portal have been evaluated repeatedly and adapted so that identified barriers for implementation for this determinant have been addressed [12, 13]. For example, PROMs are now available in multiple languages and KLIK has become an adaptable system to meet many individual wishes of the multidisciplinary teams [11]. However, little is known about barriers at the level of both clinicians and patients/parents. More insight is needed to fully understand the experienced barriers and to be able to optimize the KLIK PROM portal with regard to the wishes and needs of the user. Therefore, the aim of this study is to get more systematic insight into the experiences with KLIK from a clinician’s perspective.
Methods
KLIK implementation process
KLIK can be implemented for every multidisciplinary team (e.g., diabetes, dermatology) in health care [12]. The implementation process starts at request of a multidisciplinary team and is guided by the KLIK expert team (consisting of researchers with expertise in the field of PROMs and HRQOL research) of the Emma Children’s Hospital Amsterdam UMC through the following phases (Fig. 2):
-
1.
The KLIK expert team has an exploratory meeting with the clinicians of the multidisciplinary team to get an impression of the patient group and the Patient Reported Outcomes (PROs) they would like to discuss in the consultation room.
-
2.
The KLIK expert team gives advice about reliable, sensitive, and valid PROMs to measure the desired PROs. Whenever possible, PROMs with high reliability for specific populations and settings are selected to be able to use them on an individual level. However, sometimes the psychometric properties are not sufficient or unknown for the specific population, but no alternatives are available (e.g., in pediatrics, or in rare diseases).
-
3.
The KLIK website is set up according to the wishes and workflow of the multidisciplinary team (e.g., frequency of completing PROMs, which reminder e-mails should be sent etc.). At this moment, over 300 PROMs have been built into the KLIK PROM portal. PROMs are offered to patients depending on age and patient group. Each member of the multidisciplinary team sees feedback of their preferred outcome measure set in a personal KLIK ePROfile.
-
4.
Prior to the start of the implementation, all clinicians are trained in the use of KLIK in the consultation room. The 1.5 h training consists of a theoretical and a practical part. In the theoretical part, attention is paid to the definition of PROs and PROMs, the importance of discussing PROMs in the consultation room, and the use of the KLIK PROM portal including the different feedback options. In the practical part clinicians are trained in discussing the KLIK ePROfile with patients [14].
-
5.
Throughout the implementation process, the KLIK expert team acts as a helpdesk for both clinicians and patients. For example, the KLIK expert team supports the integration of KLIK into the existing workflow of a multidisciplinary team and helps patients and/or parents to log into the KLIK website and complete PROMs.
-
6.
As standard part of the KLIK implementation process, the KLIK expert team offers annual one-hour evaluation meetings to multidisciplinary teams to evaluate the use of KLIK in daily clinical practice and to identify and overcome barriers in the implementation process.
Design and procedure
From February 2018 until August 2019, online evaluation questionnaires were sent out one week prior to each evaluation meeting. Reminder e-mails were sent to clinicians who had not completed the questionnaire one day before the meeting. The answers of the clinicians on the questionnaire on a team level provided a starting point for the evaluation meeting. Clinicians who had not completed the questionnaire prior to this meeting were asked to do so afterwards. This study has been approved by the Medical Ethics Committee of the Amsterdam University Medical Centers (Amsterdam UMC).
Participants
Two hundred and forty-three team members (independent of their presence during the evaluation meeting) of 36 multidisciplinary teams in 14 hospitals that use KLIK were approached to participate in this study prior to a KLIK evaluation meeting. Multidisciplinary teams who use the KLIK PROM portal only for scientific purposes (6 multidisciplinary teams), where the implementation process started less than a year ago (N = 14) or teams that did not respond (N = 14) were not eligible. Supplement 1 provides an overview of the inclusion process.
Measure
An evaluation questionnaire (Supplement 2) was developed to obtain the opinion of clinicians about the use of KLIK in daily clinical practice. The evaluation questionnaire was composed by four researchers of the KLIK expert team and reviewed by three nurses and one pediatrician. The questionnaire consisted of 20 closed questions (response options: three- and five-point Likert Scales, Visual Analogue Scales (VAS) and check boxes) and four mandatory open questions (a) advantages and (b) disadvantages of KLIK, (c) incentives for patients and (d) frequently heard reactions of patients about KLIK) regarding (1) overall satisfaction, (2) feeling competent to discuss PROMs, (3) use of KLIK during the consultation, (4) influence of KLIK on the consultation, (5) usability of the KLIK PROM portal, (6) satisfaction with PROMs and feedback, and (7) support of the KLIK expert team. There was room to add a comment or explanation with each question. Since every multidisciplinary team uses a different subset of PROMs and feedback options, not all questions in the domain ‘satisfaction with PROMs and feedback’ could by answered be all clinicians.
Analysis
The Statistical Package for Social Sciences (SPSS) version 25.0 was used for descriptive statistics (percentages) to provide insight into the opinion of clinicians regarding KLIK and to study barriers and facilitators for the implementation process. Open questions of the evaluation questionnaire were analyzed qualitatively, by clustering the answers of all clinicians into main themes. This was done by two researchers (LT & HAvO) following the method for thematic analysis in Psychology [18]. Themes are ranked based on the number of times they have been mentioned by the clinicians (most often to fewest times).
Results
Participants
The online evaluation questionnaire was completed by 148 clinicians (61%), who were part of 36 different multidisciplinary teams from the following 14 different hospitals (Supplement 1): Emma Children’s Hospital (N = 57 participating clinicians), Amsterdam UMC locations VU Medical Center (N = 24) and Academic Medical Center (N = 4), Kidz & Ko – diabetes collaboration centers (N = 18), Reade (N = 8), University Medical Center Groningen (N = 7), Spaarne Hospital (N = 6), VieCuri Medical Center (N = 6), Zuyderland Medical Center (N = 5), Maasstad Hospital (N = 5), Kempenhaeghe epilepsy center (N = 3), Sophia Children’s Hospital (N = 2), Radboud University Medical Center (N = 2), and Wilhelmina Children’s Hospital (N = 1). Discipline and disease group of participating clinicians are shown in Table 1. On average, participating clinicians used KLIK for 3.3 years (range 0.2–8.8 years). Most participating clinicians were employed as medical doctor (N = 57), psychologist (N = 39), or nurse (N = 36), and multidisciplinary teams were divided into pediatrics (32 teams) and adult health care (4 teams).
Overall satisfaction
Clinicians (N = 147) reported an overall satisfaction with the KLIK PROM portal of median = 69, range 13–100, on a VAS ranging from 0 (not satisfied) to 100 (very satisfied). One clinician could not fill in the VAS due to technical problems.
Feeling competent to discuss PROMs
Almost all clinicians (89.9%) indicated that the KLIK training had prepared them sufficiently to use KLIK in daily clinical practice (8.1% neutral, 2% disagree). In addition, 85.8% of the clinicians felt competent to discuss the KLIK ePROfile with patients and/or parents in the consultation room (7.4% neutral, 6.8% disagree).
Use of KLIK during the consultation
Table 2 gives an overview of the use of KLIK reported by the clinicians. Most clinicians (70.3%) indicated they discuss the KLIK ePROfile (almost) always with patients and/or parents, 18.2% reported to discuss the KLIK ePROfile sometimes and 11.5% indicated to (almost) never discuss the KLIK ePROfile. Reasons for not discussing the KLIK ePROfile with patients and/or parents, as indicated by clinicians in the comments section, were lack of time, PROMs not completed, forgot to discuss, technical problems, no priority, no problems reported in the KLIK ePROfile, the KLIK ePROfile was discussed by another team member or KLIK was no longer part of standard care. Clinicians indicated they discuss the KLIK ePROfile at the start (42.6%), middle (37.8%), or end (19.6%) of the consultation. Clinicians estimated that they spend on average 15% of the consultation (broad range of consultation time; 10–50 min) on discussing the KLIK ePROfile and 85.8% of the clinicians were satisfied with this percentage.
The majority of the clinicians (70.3%) invited all patients to participate in the KLIK PROM portal. Patients were not invited for the following reasons: absence of a chronic health condition, presence of a language barrier, a mental disability, illiteracy or not falling into a specific age range. In addition, clinicians mentioned they sometimes forgot to invite patients or they did not see it as their responsibility. 43.2% of the clinicians estimated that 75–100% of their patients and/or parents completed the PROMs. According to clinicians, reasons for not completing PROMs by patients were no Internet access, language barrier, forgetting, and loss of motivation.
Influence of KLIK on the consultation
According to 70.3% of the clinicians, their consultation improved by the use of the KLIK PROM portal (24.3% neutral, 5.4% disagree) and 60.1% of the clinicians detected problems in functioning of patients and/or parents sooner (33.8% neutral, 6.1% disagree). Reasons for not detecting problems sooner with the use of KLIK were that another team member discussed the KLIK ePROfile with patients and/or parents or that the clinician was already aware of the functioning of the patients. Half of the clinicians (48.6%) indicated that they thought patients and/or parents were satisfied with the use of KLIK, 45.3% of the clinicians indicated that they did not know and 6.1% of the clinicians indicated that they thought patients and/or parents were not satisfied. Reasons why patients were not satisfied according to clinicians were: many questions (time intensive, having to complete PROMs too often, repetition in questions), practical problems (no Internet, login problems) and/or no motivation (annoying, no added value).
Regarding the open questions (Table 3), main advantages of KLIK for clinicians were: insight in patient’s functioning, improved communication, detecting problems, insightful feedback, patients being better prepared, easy to use, time saving, and clinician was better prepared. Main disadvantages of KLIK for clinicians were: low response-rate, takes time for clinician, irrelevant content of PROMs, complex procedure, technical aspects, no integration with Electronic Health Record (EHR), and takes time for patients. Table 3 shows the most important advantages and disadvantages of KLIK, expressed by clinicians.
According to clinicians, incentives for patients to use the KLIK PROM portal were: insight in functioning (reflection, awareness), preparation for consultation (time to think, conversation topics), improved communication (starting point for conversation, structure, comprehensive), feeling heard (being taken seriously, acknowledgement), to be offered interventions in time (signaling, intervene), and empowerment (involvement, request for help). Ten clinicians (6.8%) indicated that they do not know what the benefits for patients are.
Usability of the KLIK PROM portal
According to 71.6% of the clinicians, the KLIK portal is easy to use (19.6% neutral, 8.8% disagree) and 83.8% of the clinicians indicated that the KLIK portal has an attractive layout (15.5% neutral, 0.7% disagree).
Satisfaction with PROMs and feedback
In general, 64.9% of the clinicians were satisfied with the selected PROMs (Table 4). Reasons why clinicians were not satisfied with the PROMs were too many PROMs, PROMs are not suitable for every patient and not all PROMs are available in multiple languages. Regarding the feedback of answers of the PROMs, 80.4% of the clinicians were satisfied with the feedback in the overall KLIK ePROfile. In the KLIK ePROfile the individual items in traffic light colors (Fig. 1a) were viewed most frequently by the clinicians (84.7%). Of these traffic light colors, clinicians discussed the red answers most often with patients/parents (84.7%), followed by orange (58.4%) and green answers (34.3%). The graphs (scores over time resp. comparison with peers) are discussed by 47.4% resp. 33.6% of the clinicians. Clinicians thought that the traffic light colors of the KLIK ePROfile are most important (median = 72), followed by literal answers (median = 71) and graphs (median = 70) (Fig. 1b), reported on a VAS, ranging from 0 (not important) to 100 (very important).
Support KLIK expert team
82.5% of the clinicians indicated to know where to ask their questions regarding the use of the KLIK PROM portal (10.1% neutral, 7.4% disagree) and 71.6% indicated that there is enough support from the KLIK expert team (25.7% neutral, 2.7% disagree).
Discussion
This study provided insight into the experiences of clinicians with the use of the KLIK PROM portal in daily clinical care, at a group level. Overall, clinicians were satisfied with discussing PROMs in the consultation room via the KLK PROM portal. Clinicians indicated that discussing PROMs helps them to gain more insight into patient functioning, to improve the communication with patients, to detect psychosocial or physical problems, and to empower patients. These benefits are in line with previous effectiveness studies [3, 4, 6]. In addition, clinicians valued specific characteristics of the KLIK ePROfile, such as ease of use and the well-developed and insightful feedback. Regarding this feedback, clinicians mentioned they appreciated and looked at the individual item feedback in traffic light colors most often. This preference was also found in previous research on the feedback of the QLIC-ON Profile [19].
Although clinicians indicated that the KLIK training sufficiently prepared them to use KLIK in clinical practice, they also indicated that the training did not fully meet their needs. More explanation about the interpretation of PROM results and the use of cutoff scores would increase their sense of competence. In addition, a refresher course every few years would be desirable. For this reason, the KLIK expert team is now revising the KLIK training. More information and tips and tricks about the interpretation and communication of PROM results will be included.
Clinicians indicated that they do not always discuss the PROMs with patients and/or parents due to lack of time, technical problems or lack of clarity regarding the workflow. For some clinicians it is unclear which team member of the multidisciplinary team discusses the PROMs with patients and/or parents or who sends invitations. This indicates that continuous support with the implementation process and annual evaluation meetings with all team members of a multidisciplinary team remains necessary. Also, patients do not always complete PROMs prior to the outpatient consultation. Forgetting, loss of motivation or no Internet access were reasons from the clinicians’ perspective. In supporting the implementation process, a commonly heard argument from patients for not completing the PROMs is that the clinician does not discuss the PROMs during the consultation. This indicates how important it is for clinicians to discuss the PROMs with patients and/or parents. In addition, it was mentioned that for patients (or parents) with low health literacy skills and for non-native Dutch speakers it is sometimes difficult to complete the PROMs. Although the most frequently used generic PROMs in KLIK are available in multiple languages, this is not the case for all PROMs. When compiling the PROMs outcome sets with the multidisciplinary teams, more attention should also be paid to the needs of non-native Dutch speakers and patients with low health literacy skills.
Clinicians reported several main barriers for using PROMs via the KLIK portal. The first one is a lack of integration between KLIK and the EHRs. Opening a separate website to view the KLIK ePROfile is an added operation for clinicians, with the consequence that the KLIK ePROfile is sometimes not discussed with patients and/or parents. Therefore, in September 2019 a front-end integration with the two most often used EHRs in the Netherlands, Epic© and HiX© was realized in four hospitals. Clinicians can now view the KLIK ePROfile via the EHR, which increases the user-friendliness and makes it a better fit into the clinical workflow.
Second, clinicians indicated that they are not always satisfied with the content of PROMs. Reasons were mostly focused on the burden of completing PROMs for patients, such as a long completion time, many repetitions in questions and irrelevant questions. These challenges with PROMs correspond with previous research [20]. To address these problems, the National Institute of Health (NIH) developed the Patient-Reported Outcome Measurement Information System (PROMIS) [21, 22]. PROMIS consists of various dynamic item banks (each measuring a separate construct) that can be administered through computerized adaptive testing (CAT) [20, 23]. By using a CAT, questions are offered based on the person’s previous answer. In this way, patients and/or parents only have to answer a few questions per PROMIS construct to get a reliable score. As a result, the burden for patients and/or parents can be reduced [24]. Since November 2019, it is possible to administer the PROMIS item banks via KLIK, by linking KLIK with the Dutch Assessment Center. To realize this, the PROMIS item banks were translated and validated in the Netherlands [11, 16].
Third, clinicians mentioned that the use of PROMs is time intensive. Clinicians indicated that it takes more time to prepare themselves for the consultation and to discuss the PROMs in the consultation room. This is a remarkable finding, since previous research has shown that the use of the QLIC-ON Profile did not lengthen the consultation [9]. In addition, clinicians who are responsible for inviting patients for the KLIK PROM portal indicated that it takes a lot of effort to motivate patients to complete PROMs. A case manager that supports the KLIK implementation would be helpful.
There were a few limitations to this study. First, not all clinicians that use KLIK in the consultation room have been included in this study, because not all multidisciplinary teams were open to an evaluation meeting despite the importance for the implementation process. However, the experiences of clinicians from different disciplines, working with various disease groups in multiple hospitals and different outcome measure sets were included. Second, completing the VAS of the domains ‘overall satisfaction’ and ‘satisfaction with PROMs and feedback’ was not always possible when using a tablet. For these clinicians, it was not possible to move the bar to the desired position, causing a score around 50. Unfortunately, it could not be traced who had had this problem and therefore the results of these questions should be interpreted carefully. Third, the question ‘I am satisfied with the PROMs offered’ was not always understood by the clinicians. Prior to this question, there was a question about specific PROMs. The explanations showed that some clinicians referred to the specific PROMs when answering this question. That is why the answers to this question of 14 clinicians were not included. Fourth, due to the used method, this study provides no insight into the actions clinicians take with regard to the completed PROMs. In addition, no questions were asked about how clinicians use the information from the completed PROMs in daily clinical care. Therefore, recommendations for future PROM implementation research are to gain more insight into the actions of clinicians with regard to the discussed PROMs and how this can lead to more patient-centered care. The use of video observations in the consultation room may provide this information.
To conclude, the KLIK PROM portal is a valuable tool for clinicians to systematically monitor the functioning of their patients in clinical practice, so that extra support can be offered when needed. Overall, clinicians were enthusiastic about the feedback and user-friendliness of the KLIK PROM portal and the added value of using PROMs in clinical practice. However, some challenges and barriers were also identified. Therefore, a next step is to address the mentioned feedback points in the KLIK portal to improve the user-friendliness. Also, the perspective of the other user group, the patients and parents, is needed to further adapt the KLIK PROM portal to their wishes. Therefore, a similar study will be performed in the near future evaluating the KLIK PROM portal from the patients’ perspective, with the ultimate goal to further optimize the KLIK PROM portal and to improve the quality of health care.
References
Santana, M. J., & Feeny, D. (2014). Framework to assess the effects of using patient-reported outcome measures in chronic care management. Quality of Life Research, 23(5), 1505–1513. https://doi.org/10.1007/s11136-013-0596-1.
Bouazza, Y. B., Chiairi, I., El Kharbouchi, O., De Backer, L., Vanhoutte, G., Janssens, A., et al. (2017). Patient-reported outcome measures (PROMs) in the management of lung cancer: A systematic review. Lung Cancer, 113, 140–151. https://doi.org/10.1016/j.lungcan.2017.09.011.
Valderas, J. M., Kotzeva, A., Espallargues, M., Guyatt, G., Ferrans, C. E., Halyard, M. Y., et al. (2008). The impact of measuring patient-reported outcomes in clinical practice: A systematic review of the literature. Quality of Life Research, 17(2), 179–193. https://doi.org/10.1007/s11136-007-9295-0.
Marshall, S., Haywood, K., & Fitzpatrick, R. (2006). Impact of patient-reported outcome measures on routine practice: A structured review. Journal of Evaluation in Clinical Practice, 12(5), 559–568. https://doi.org/10.1111/j.1365-2753.2006.00650.x.
Greenhalgh, J., Gooding, K., Gibbons, E., Dalkin, S., Wright, J., Valderas, J., et al. (2018). How do patient reported outcome measures (PROMs) support clinician-patient communication and patient care? A realist synthesis. J Patient Rep Outcomes, 2, 42. https://doi.org/10.1186/s41687-018-0061-6.
Anthony, S. J., Stinson, H., Lazor, T., Young, K., Hundert, A., Santana, M. J., et al. (2019). Patient-reported outcome measures within pediatric solid organ transplantation: A systematic review. Pediatric Transplantation. https://doi.org/10.1111/petr.13518.
Basch, E., Deal, A. M., Dueck, A. C., Scher, H. I., Kris, M. G., Hudis, C., et al. (2017). Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA, 318(2), 197–198. https://doi.org/10.1001/jama.2017.7156.
de Wit, M., Delemarre-van de Waal, H. A., Bokma, J. A., Haasnoot, K., Houdijk, M. C., Gemke, R. J., et al. (2008). Monitoring and discussing health-related quality of life in adolescents with type 1 diabetes improve psychosocial well-being: A randomized controlled trial. Diabetes Care, 31(8), 1521–1526. https://doi.org/10.2337/dc08-0394.
Engelen, V., Detmar, S., Koopman, H., Maurice-Stam, H., Caron, H., Hoogerbrugge, P., et al. (2011). Reporting health-related quality of life scores to physicians during routine follow-up visits of pediatric oncology patients: Is it effective? Pediatric Blood & Cancer, 58(5), 766–774. https://doi.org/10.1002/pbc.23158.
Haverman, L., Van Rossum, M. A., Van Veenendaal, M., van den Berg, J. M., Dolman, K. M., Swart, J., et al. (2013). Effectiveness of a web-based application to monitor health-related quality of life. Pediatrics, 131(2), 533–543. https://doi.org/10.1542/peds.2012-0958.
Haverman, L., van Oers, H. A., van Muilekom, M. M., & Grootenhuis, M. A. (2019). Options for the interpretation of and recommendations for acting on different PROMs in daily clinical practice using KLIK. Medical Care, 57(Suppl5 Suppl 1), S52–S58. https://doi.org/10.1097/MLR.0000000000001061.
Haverman, L., van Oers, H. A., Limperg, P. F., Hijmans, C. T., Schepers, S. A., Sint Nicolaas, S. M., et al. (2014). Implementation of electronic patient reported outcomes in pediatric daily clinical practice: The KLIK experience. Clinical Practice in Pediatric Psychology, 2(1), 50–67.
Schepers, S. A., Sint Nicolaas, S. M., Haverman, L., Wensing, M., van Meeteren, A. Y., Veening, M. A., et al. (2016). Real-world implementation of electronic patient-reported outcomes in outpatient pediatric cancer care. Psychooncology, 26(7), 951–959. https://doi.org/10.1002/pon.4242.
Santana, M. J., Haverman, L., Absolom, K., Takeuchi, E., Feeny, D., Grootenhuis, M., et al. (2015). Training clinicians in how to use patient-reported outcome measures in routine clinical practice. Quality of Life Research, 24(7), 1707–1718.
van Oers, H. A., Schepers, S. A., Grootenhuis, M. A., & Haverman, L. (2016). Dutch normative data and psychometric properties for the distress thermometer for parents. Quality of Life Research, 26(1), 177–182. https://doi.org/10.1007/s11136-016-1405-4.
Haverman, L., Grootenhuis, M. A., Raat, H., van Rossum, M. A., van Dulmen-den Broeder, E., Hoppenbrouwers, K., et al. (2016). Dutch-Flemish translation of nine pediatric item banks from the patient-reported outcomes measurement information system (PROMIS)(R). Quality of Life Research, 25(3), 761–765. https://doi.org/10.1007/s11136-015-0966-y.
CBS. (2018). The Netherlands leads Europe in internet access. Accessed February 12 2020.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.
Engelen, V., Haverman, L., Koopman, H., Schouten-van Meeteren, N., Meijer-van den Bergh, E., Vrijmoet-Wiersma, J., et al. (2010). Development and implementation of a patient reported outcome intervention (QLIC-ON PROfile) in clinical paediatric oncology practice. Patient Education and Counseling, 81(2), 235–244. https://doi.org/10.1016/j.pec.2010.02.003.
Terwee, C., Roorda, L., De Vet, H., Dekker, J., Westhovens, R., Van Leeuwen, J., et al. (2014). Dutch-Flemish translation of 17 item banks from the patient-reported outcomes measurement information system (PROMIS). Quality of Life Research, 23(6), 1733–1741.
Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., Yount, S., et al. (2010). The patient-reported outcomes measurement information system (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology, 63(11), 1179–1194. https://doi.org/10.1016/j.jclinepi.2010.04.011.
Cella, D., Yount, S., Rothrock, N., Gershon, R., Cook, K., Reeve, B., et al. (2007). The patient-reported outcomes measurement information system (PROMIS): Progress of an NIH Roadmap cooperative group during its first two years. Medical Care, 45(5 Suppl 1), S3–s11. https://doi.org/10.1097/01.mlr.0000258615.42478.55.
Reeve, B. B., Hays, R. D., Bjorner, J. B., Cook, K. F., Crane, P. K., Teresi, J. A., et al. (2007). Psychometric evaluation and calibration of health-related quality of life item banks: Plans for the patient-reported outcomes measurement information system (PROMIS). Medical Care, 45(5 Suppl 1), S22–31. https://doi.org/10.1097/01.mlr.0000250483.85507.04.
Cella, D., Gershon, R., Lai, J. S., & Choi, S. (2007). The future of outcomes measurement: Item banking, tailored short-forms, and computerized adaptive assessment. Quality of Life Research, 16(Suppl 1), 133–141. https://doi.org/10.1007/s11136-007-9204-6.
Acknowledgements
The authors would like to thank all clinicians that participated in this study and Biomedia for their support in sending out the evaluation questionnaire.
Author information
Authors and Affiliations
Contributions
LT, MMvM, LH and HAvO developed the evaluation questionnaire. LT, AG and HAvO carried out the data collection. LT, LK and HAvO performed the statistical analysis. The first draft of the manuscript was written by LT. JBvG, MAG, LH and HAvO handled the supervision. All authors critically revised the manuscript for intellectual content and approved the final version of the manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
All procedures performed in this study were in accordance with the ethical standards of the international and/or national research committee (Medical Ethics Committee of the Amsterdam UMC – W18_091 # 18.117) and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed consent
Informed consent was obtained from all individual participants included in this study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Teela, L., van Muilekom, M.M., Kooij, L.H. et al. Clinicians’ perspective on the implemented KLIK PROM portal in clinical practice. Qual Life Res 30, 3267–3277 (2021). https://doi.org/10.1007/s11136-020-02522-5
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11136-020-02522-5