The KLIK Patient Reported Outcome Measures (PROM) portal (www.hetklikt.nu) has been implemented since 2011 in clinical practice in over 20 Dutch hospitals. Patients and/or parents complete PROMs before the outpatient consultation and answers are subsequently discussed by clinicians during consultation. This study aims to provide insight into patients’ and parents’ perspective on the use of the KLIK PROM portal in order to optimize its implementation in pediatric clinical practice.
Patients (12–19 years) and parents (of children 0–19 years) from the Emma Children’s Hospital were invited to participate. A mixed-method design was used; (1) Focus groups were held and analyzed using thematic analysis in psychology, (2) a questionnaire was sent out and analyzed using descriptive statistics.
(1) Eight patients and 17 parents participated. Patients mentioned that KLIK has an attractive layout. However, PROMs were sometimes considered irrelevant and repetitive. Parents valued that KLIK provides insight into their child’s functioning, but they were not satisfied with the extent to which PROMs were discussed by clinicians. (2) 31 patients and 130 parents completed the questionnaire. Overall, patients and parents reported a satisfaction score of 7.9/10 and 7.3/10, respectively. 81% of patients and 74% of parents indicated that KLIK is easy to use.
Patients and parents are generally satisfied with KLIK, however, points of improvement were mentioned. These are currently being addressed by e.g., upgrading the KLIK website, implementing PROMIS item banks in KLIK to reduce irrelevancy and repetitiveness of PROMs, and implementation strategies to improve the discussion-rate. In this way, implementation of the KLIK PROM portal can be further optimized, with the ultimate goal to improve quality of care.
Patient Reported Outcome Measures (PROMs) are increasingly used to monitor and discuss symptoms, Health-Related Quality of Life (HRQOL) and psychosocial functioning of patients in the consultation room with the ultimate goal to enable shared-decision making and patient-centered care [1,2,3]. Using PROMs in clinical practice has been shown valuable, as it results in more awareness for and increased discussion of patient concerns, higher patient satisfaction, better communication between patient and clinician, and improved treatment outcomes [4,5,6,7,8,9].
A system that facilitates the use of PROMs in clinical practice is the evidence-based KLIK PROM portal (www.hetklikt.nu) [10,11,12,13], which has been implemented in over 20 hospitals in the Netherlands since 2011 . With KLIK, pediatric patients and/or their parents, and adult patients complete PROMs before the outpatient consultation. Answers are converted into an electronic KLIK PROfile (KLIK ePROfile) which the clinician discusses with patients and parents during the consultation . The most important stakeholders in the development and implementation process of the KLIK PROM portal are the users; clinicians as well as patients/parents. From the onset of KLIK, clinicians’ opinions were asked during these processes. For example, clinicians’ preferences for PROM feedback options in the KLIK ePROfile were studied , clinicians were involved in the selection of PROs and PROMs for their disease group, and they were consulted in annual evaluation meetings to identify and overcome barriers in the implementation process . Two studies were performed to gain more insight into the experiences of clinicians with KLIK and to identify barriers in the implementation process, with the goal to improve the KLIK PROM portal according to their needs [15, 16]. However, the opinion of the other stakeholder, patient/parents, is also important , as engaging patients in KLIK could result in higher patient satisfaction and higher enrollment rates [18,19,20,21].
Worldwide, patients are increasingly engaged in PROM development (e.g., item development, comprehensibility)  and PROM visualization to patients and clinicians . However, the experiences of patients regarding the use of PROMs in daily clinical practice has received less consideration [24,25,26,27,28,29,30,31]. Available studies explored the experiences of adult patients regarding the use of PROMs in daily clinical practice. Both positive (e.g., improved communication, insight into patient’s functioning, and increased awareness of psychosocial problems) [25, 26, 28,29,30,31] and negative experiences (e.g., negative and irrelevant questions in PROMs, unclear purpose of using PROMs) [25,26,27] were identified. To our knowledge, no studies have been performed focusing on the experiences of pediatric patients and their parents with using PROMs in daily clinical practice. To be able to optimize and further implement the KLIK PROM portal, it is also necessary to gain understanding of their wishes and needs. Therefore, the aim of this study is to provide more insight into the perspective of patients and parents on the implementation of PROMs in pediatric clinical practice using the KLIK PROM portal.
The KLIK workflow for pediatric patients and parents consists of several steps; (1) creation of a KLIK account by patients/parents, (2) completion of PROMs by patients/parents before the outpatient consultation, (3) conversion of answers into a KLIK ePROfile, and (4) discussion of the KLIK ePROfile by the clinician during consultation (Fig. 1).
This study is part of a larger participation study where KLIK users’ (patients/parents) opinion was asked about several aspects of health care and the use of the KLIK PROM portal. This sub-study reports on the evaluation of the KLIK PROM portal. A mixed-method design was used where qualitative and quantitative methodologies were combined: (1) focus groups were held with patients and parents and (2) an evaluation questionnaire was sent out to pediatric patients and parents. The Medical Ethics Committee of the Amsterdam University Medical Centers (Amsterdam UMC—AMC) approved this study. All participants provided informed consent.
Patients (12–19 years) and parents (of children 0–19 years) who consult a pediatric department of the Emma Children’s Hospital Amsterdam UMC that uses KLIK as standard part of care, completed KLIK PROMs at least once (questionnaire) or twice (focus groups), and were part of the ‘KLIK panel’ could participate in this mixed-method study. Patients with any chronic health condition could participate in this study as the workflow of the KLIK PROM portal is similar for all patient groups. The ‘KLIK panel’ consists of patients and parents that indicated, during registration on the KLIK PROM portal, that they give permission to be invited for research projects. Eligible patients/parents were invited by e-mail to take part in the focus groups (March 2018) and/or to complete the evaluation questionnaire (June–December 2019). Socio-demographics (age and gender child), information on chronic health condition of the child and years of using KLIK were obtained from the KLIK PROM portal. All participants received a gift card of 5 euros (focus groups) or 10 euros (questionnaire) after participation.
Focus groups with patients and parents were held separately and for each focus group inclusion of three to six participants was pursued . Focus groups consisted of a group discussion guided by two moderators (MvM, LT, HvO, or LH). At the start of the focus group, the aim of the study was explained and a short recapitulation of KLIK was provided. Then, to obtain patients’ and parents’ opinion about KLIK, positive and negative experiences with KLIK were discussed using the evaluation technique ‘Complain and Cheer wall’ . Participants were asked to write down their positive experiences on a flip over at one side of the room, what we called the ‘Cheer wall’, and points of improvement on another flip over at the other side of the room, the ‘Complain wall’. Thereafter a group discussion took place and topics on the walls were grouped together into main themes. Duration of each focus group was 60 min. All focus groups were audio recorded.
The questionnaire (separate version for patients and parents, with minor differences regarding language use—Supplement 1) was developed by five researchers of the KLIK expert team and reviewed by five other researchers and one psychologist. Both versions of the questionnaire consisted of 17 closed questions (response options: three- and five-point Likert Scales and Visual Analogue Scales (VAS)) and two mandatory open questions (advantages and disadvantages of KLIK), regarding (1) overall satisfaction with the KLIK PROM portal, (2) completion of PROMs in the KLIK PROM portal, (3) discussing PROMs with the clinician, (4) influence of KLIK on the (preparation of) the consultation, (5) usability of the KLIK PROM portal, and (6) content of PROMs. For three closed questions, an additional mandatory open question was provided, asking about the reason for their answer.
Descriptive analyses were performed using the Statistical Package for Social Sciences (SPSS) version 25.0 to characterize the participants.
Regarding the focus groups, all audio recordings were transcribed verbatim and the transcripts were analyzed independently by MvM and LT in MAXQDA (2018) following the thematic analysis in psychology : (1) highlighting relevant parts of the manuscript, (2) organizing data into meaningful groups by generating initial codes, (3) collating initial codes into themes, (4) refining themes into main- and subthemes, (5) defining the final themes. Analyses were discussed until consensus was reached on the themes. Data saturation was considered attained when no new themes emerged during the analyses of the focus groups.
Regarding the questionnaire, SPSS was used for descriptive statistics (percentages) to provide insight into the experiences of patients and parents with the use of the KLIK PROM portal. Open questions of the evaluation questionnaires were analyzed qualitatively by MvM and LT. This was done by clustering the answers of both patients and parents into main themes following the thematic analysis in psychology .
Figure 2 shows the study and participant flowchart of this study. In total, 8 patients (three focus groups) and 17 parents (three focus groups) participated in six focus groups. Regarding the questionnaire, 31 patients (response rate: 21.8%) and 130 parents (response rate: 19.6%) participated. One patient and 5 parents participated in the focus groups and completed the questionnaire. Table 1 shows the sociodemographic characteristics of all participants.
Data saturation was attained as no new themes emerged after analyzing the focus groups. Table 2 (patients) and 3 (parents) depict the most important positive experiences with KLIK and points of improvement for KLIK and corresponding examples of statements. Themes are ranked based on the number of times mentioned (most often to fewest times) by patients and parents during the focus groups.
In all focus groups, patients came up with a broad range of experiences with KLIK, both positive, negative and mixed. Themes that were unanimously rated as positive were that the KLIK website has an attractive layout (due to the use of colors and pictures), that KLIK provides insight into their daily functioning and that KLIK improves the conversation content during the consultation, where a broader range of topics is discussed. Furthermore, patients indicated that the consultation is more efficient when using KLIK and that they are happy about how secure the KLIK website is and how their data remains anonymous. There were five themes on which patients disagreed. Some patients rated the content of PROMs positively, as they cover all important topics and are clear, while other patients indicated that the questions in the PROMs are difficult to understand, repetitive and not relevant for every patient. In addition, completion time was rated by some as good and by others as time-consuming, and the KLIK ePROfile is always discussed by the clinician according to some patients, but not enough by others. Finally, KLIK helps only some patients in preparing for the consultation, and patients were ambiguous about ease of use of KLIK. The lack of motivation for completing the KLIK PROMs was only mentioned as a negative experience by some patients.
Parents mentioned many similar experiences with KLIK as patients (Table 3). Themes that were unanimously rated as positive were that KLIK helps in preparing for the consultation and provides insight into the patients’ functioning, although for some parents this insight was also confronting when many problems were reported. In addition, parents were satisfied that by using KLIK problems are detected at an early stage and that support can be provided timely. All other themes were evaluated both positively and negatively. Some parents indicated that they are satisfied with the content of PROMs, as all topics are covered and questions are easy to understand, while other parents disagreed and indicated that questions are hard to understand for their child, are confronting and repetitive. Parents also had mixed opinions regarding ease of use of KLIK, where some thought completing PROMs online is working great, and others thought this could be improved by developing a KLIK app and linking KLIK to the Electronic Health Records (EHR). Furthermore, discussion of the KLIK ePROfile by clinicians always happens according to some parents, but not often enough by even more parents. Most parents mentioned that the conversation content improves as more and different topics are discussed, while some did not recognize this. Completion time is manageable for some, but too long for others and the layout of the KLIK website is attractive and child-friendly according to most parents, but could be made more attractive by using visuals according to some parents. Finally, some parents indicated that they do not see the added value and goal of KLIK, while others disagreed and indicated that KLIK is of great value to the consultation.
Overall satisfaction with the KLIK PROM portal
Patients and parents reported an overall satisfaction with the KLIK PROM portal of mean = 7.9 and mean = 7.3, respectively, on a VAS ranging from 0 (not satisfied) to 10 (very satisfied).
Completion of PROMs in the KLIK PROM portal
As shown in Table 4, 78% of the patients and 84% of the parents agreed that they know why there are asked to complete PROMs via the KLIK PROM portal. Patients and parents reported that the frequency in which they are asked to complete these PROMs varies from once every three years to more than four times a year. Most patients and parents were satisfied with this frequency. When patients and parents are asked to complete PROMs, the majority indicated that they almost always do this. Reasons for not completing the PROMs were: lack of time, forgot to complete, little change in functioning since the last PROM completion, and no motivation. Patients and parents spent on average 13.8 and 15.2 min on completing the PROMs, respectively. More than 80% of both patients and parents were satisfied with this completion time.
Discussing PROMs with the clinician
About half of the patients and parents indicated that their clinician (almost) always discusses the KLIK ePROfile with them during the consultation (Fig. 3). If the clinician does not discuss the completed PROMs, 52% of the patients and 72% of the parents indicated they dare to start the discussion about PROMs themselves.
Influence of KLIK on the (preparation of the) consultation
KLIK is of added value for the conversation with their clinician, according to 58% of the patients and 59% of the parents (Fig. 3). Less than half of the patients and parents indicated that more topics are discussed by using the KLIK PROM portal in comparison with not using the KLIK PROM portal and that the use of KLIK provides more structure to the conversation. Clinicians’ failure to discuss the KLIK ePROfile was a frequently mentioned reason why KLIK has no value during the consultation. More than half of the parents reported that the use of KLIK provides them more insight into the functioning of their child and helps in preparing for the consultation (62% and 54% respectively), in contrast to only 39% and 42% of the patients. Patients indicated that they know very well how they are doing, even without completing a PROM.
Table 5 shows the most important advantages and disadvantages of KLIK, as reported in the open questions. The themes are ranked based on the number of times mentioned by patients and parents in the open-ended questions. Main advantages of KLIK for patients and parents were: easy to use, clinician is better prepared, patients and parents are better prepared, and insight into functioning (of my child). Main disadvantages of KLIK for patients and parents were: not easy to use, irrelevant content of PROMs, and takes time. Eleven patients (35%) and 48 parents (37%) did not experience any disadvantages with using the KLIK PROM portal.
Usability of the KLIK PROM portal
The KLIK PROM portal is easy to use, according to 81% of the patients (13% neutral and 6% disagree) and 74% of the parents (18% neutral, 8% disagree). In addition, 48% of the patients (39% neutral, 12% disagree) and 55% of the parents (36% neutral, 9% disagree) indicated that KLIK has an attractive layout.
Content of PROMs
Most patients and parents are satisfied with the PROMs they are asked to complete (Fig. 3). Almost all participants indicated that they understand the questions asked in the PROMs. Reasons why patients and parents are not satisfied with the offered PROMs were that the questions in the PROMs do not apply to them or their child, PROMs are too generic, the different questions are very similar, and the PROMs are too long. Some of the patients and parents felt that the offered PROMs do not cover all topics that are important for them. For example they miss topics like growth, parenting support, and side jobs.
This study provided insight into the experiences of patients and parents with the implementation of PROMs in pediatric clinical practice using the KLIK PROM portal. Overall, patients and parents were satisfied with the use of KLIK. They indicated that KLIK provides insight into the patient’s functioning, helps parents and clinicians in preparing for the consultation, is easy to use, and results in discussion of a broad range of topics (e.g., from disease-specific to psychosocial functioning) during the consultation. However, points of improvement were indicated regarding the content of PROMs, the layout of the KLIK PROM portal, and the discussion of PROMs by the clinician. The results described in this study are in line with previous studies [15, 25, 26].
Although patients and parents responded to the closed question of the evaluation questionnaire that they are generally satisfied with the offered PROMs in KLIK, they mentioned in the focus groups and open-ended questions that the content of PROMs is the most important point of improvement. For example, they indicated that there is repetition in questions, that irrelevant questions are administered, and that the completion time is long, resulting in a burden of completing PROMs. These challenges with PROMs have been mentioned in previous research [16, 35, 36]. To address these challenges, the self-report and proxy-versions of the Patient-Reported Outcomes Measurement Information System (PROMIS®) item banks [37,38,39] were implemented in the KLIK PROM portal in the past year and are currently used in several clinics [16, 40, 41]. The PROMIS item banks each measure a separate construct that can be administered using Computerized Adaptive Testing (CAT). With CAT, questions are presented to patients based on their previous responses. Hence, patients only have to answer a small number of questions per item bank to obtain a reliable score  and have to answer less irrelevant questions. Consequently, the burden of completing PROMs can be reduced.
Another difference between the focus groups and the questionnaire was the rating of the ease of use of the KLIK PROM portal. While in the questionnaire the majority of participants indicated that KLIK is easy to use, in the focus groups especially parents had quite some remarks on how the ease of use could be improved. Parents mentioned that an app would be a valuable addition to the KLIK website in order to complete PROMs on your mobile phone. Additionally, they would like an integration of KLIK with the EHR so that appointments are automatically linked to KLIK by which PROMs are directly available. To address these suggestions, we made the KLIK PROM portal adaptable for mobile phone use, and realized a front-end (hybrid) integration with the EHR in 2019. With this integration, clinicians can now view the KLIK ePROfile in the EHR and discuss the PROMs more easily. However, to be able to automatically link the appointments to KLIK, a full integration is necessary, which can hopefully be realized in the future.
A final difference between the focus group and questionnaire outcomes was the satisfaction with the layout of the KLIK PROM portal, which was mainly mentioned as a point of improvement in the questionnaire. Patients and parents indicated that the website looks a bit old-fashioned and could be made more attractive by using visuals. For this reason, the homepage of the KLIK website was upgraded recently. The design of the website was changed (e.g., by using visuals and creating a more professional look). In addition, specific information pages are now available for all KLIK users (pediatric patients, parents, adult patients, and clinicians).
Patients and parents mentioned in both the focus groups as the questionnaires that clinicians often do not discuss PROMs during the consultation. This is worrisome, as patients and parents indicated that this is an important reason why KLIK sometimes has no added value for the consultation which consequently may lead to loss of motivation to complete KLIK PROMs. To improve this discussion rate, several implementation strategies were used. For example, the KLIK expert team revised the KLIK training in which more attention is now paid to the importance of discussing PROMs  and this topic is discussed more thoroughly during annual evaluation meetings with clinicians , with the goal to increase their knowledge, awareness and confidence in discussing PROMs. Additionally, finding champions for each multidisciplinary team to motivate clinicians to use and discuss KLIK PROMs would be beneficial as this was identified as the most important implementation strategy in two KLIK studies [15, 17]. When clinicians do not discuss the completed PROMs, patients and some parents indicated that they do not dare to bring up for them important themes themselves. To empower patients/parents and increase their self-efficacy, educational videos were developed and made available on the KLIK homepage (article in preparation). In these videos tips and tricks are provided how patients and parents can prepare themselves for the consultation and bring up topics they want to discuss with the clinician.
When comparing this study with the KLIK evaluation study with clinicians , similar experiences regarding the KLIK PROM portal were mentioned. For example, insight into patients’ functioning, improved communication, and better preparation of the consultation were positive points they agreed on, and content of PROMs was the most important point of improvement mentioned by both user groups. However, patients/parents and clinicians mentioned a different PROM completion rate. Patients and parents indicated a very high completion rate, whereas clinicians estimated that this completion rate is much lower and that it takes a lot of effort to motivate patients to complete PROMs . A possible reason for this difference might be a bias in the current sample, as only patients and parents that were part of the KLIK panel were invited for participation. These patients/parents might be more assertive in comparison to the other KLIK users, which might have resulted in an overestimation of the PROM completion rate. Therefore, continuous support and explanation about the goal of the use of KLIK remains very important to both user groups.
There are some limitations to this study that should be mentioned. First, there was a low response rate in the evaluation questionnaire (around 20%) which was unexpected as this questionnaire was sent to participants of the KLIK panel (who indicated that they were willing to be invited for research projects). Possible reasons for the low response rate might be that (1) the willingness of patients and parents has changed as participation in the KLIK panel was only asked during registration, (2) patients and parents do not actively use the KLIK PROM portal anymore, or (3) patients and parents might be tired of completing surveys. Second, it was also difficult to motivate patients to participate in the focus groups. This resulted in a small number of participants per patient focus group (2 to 3 participants) with two moderators, which may have influenced the dynamics. Additionally, we noticed that pediatric patients found it very difficult to formulate and express their opinion and needed a lot of guidance which could have led to a bias in the results. Third, we used a self-developed questionnaire which makes comparisons with other evaluation studies difficult. However, other studies also made use of self-conducted questionnaires  or adapted questionnaires from prior studies [29,30,31], as the questions needed to be specific about features of the tool used.
In conclusion, pediatric patients and parents were satisfied with the usability and effect of the KLIK PROM portal in clinical care. KLIK provides them insight into their functioning and helps them to communicate with the clinician. However, some points of improvement were also identified, which are currently being addressed. We now have insight into the experiences of the most important stakeholders (patients/parents and clinicians) of KLIK. In the future it is important to continuously evaluate the use of the KLIK PROM portal with all stakeholders (including adult patients) to match their needs. In this way, we can further optimize and implement the KLIK PROM portal in clinical care with the ultimate goal to improve the quality of care.
The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.
Reeve, B. B., Wyrwich, K. W., Wu, A. W., Velikova, G., Terwee, C. B., Snyder, C. F., et al. (2013). ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Quality of Life Research, 22(8), 1889–1905.
Kotronoulas, G., Kearney, N., Maguire, R., Harrow, A., Di Domenico, D., Croy, S., et al. (2014). What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials. Journal of Clinical Oncology, 32(14), 1480–1501. https://doi.org/10.1200/jco.2013.53.5948
Santana, M. J., & Feeny, D. (2014). Framework to assess the effects of using patient-reported outcome measures in chronic care management. Quality of Life Research, 23(5), 1505–1513. https://doi.org/10.1007/s11136-013-0596-1
Valderas, J. M., Kotzeva, A., Espallargues, M., Guyatt, G., Ferrans, C. E., Halyard, M. Y., et al. (2008). The impact of measuring patient-reported outcomes in clinical practice: A systematic review of the literature. Quality of Life Research, 17(2), 179–193. https://doi.org/10.1007/s11136-007-9295-0
Marshall, S., Haywood, K., & Fitzpatrick, R. (2006). Impact of patient-reported outcome measures on routine practice: A structured review. Journal of Evaluation in Clinical Practice, 12(5), 559–568. https://doi.org/10.1111/j.1365-2753.2006.00650.x
Basch, E., Deal, A. M., Dueck, A. C., Scher, H. I., Kris, M. G., Hudis, C., et al. (2017). Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA, 318(2), 197–198. https://doi.org/10.1001/jama.2017.7156
de Wit, M., Delemarre-van de Waal, H. A., Bokma, J. A., Haasnoot, K., Houdijk, M. C., Gemke, R. J., et al. (2008). Monitoring and discussing health-related quality of life in adolescents with type 1 diabetes improve psychosocial well-being: a randomized controlled trial. Diabetes Care, 31(8), 1521–1526. https://doi.org/10.2337/dc08-0394
Cheng, L., Kang, Q., Wang, Y., & Hinds, P. S. (2020). Determining the effectiveness of using patient-reported outcomes in pediatric clinical practices. Journal of Pediatric Nursing., 55, 100–109. https://doi.org/10.1016/j.pedn.2020.07.005
Bele, S., Chugh, A., Mohamed, B., Teela, L., Haverman, L., & Santana, M. J. (2020). Patient-reported Outcome measures in routine pediatric clinical care: A systematic review. Frontiers in Pediatrics, 8, 364. https://doi.org/10.3389/fped.2020.00364
Engelen, V., Haverman, L., Koopman, H., Schouten-van Meeteren, N., Meijer-van den Bergh, E., Vrijmoet-Wiersma, J., et al. (2010). Development and implementation of a patient reported outcome intervention (QLIC-ON PROfile) in clinical paediatric oncology practice. Patient Education and Counseling., 81(2), 235–244. https://doi.org/10.1016/j.pec.2010.02.003
Haverman, L., Engelen, V., Van Rossum, M. A., Heymans, H. S., & Grootenhuis, M. A. (2011). Monitoring health-related quality of life in paediatric practice: Development of an innovative web-based application. BMC Pediatrics, 11, 3–10. https://doi.org/10.1186/1471-2431-11-3
Engelen, V., Detmar, S., Koopman, H., Maurice-Stam, H., Caron, H., Hoogerbrugge, P., et al. (2011). Reporting health-related quality of life scores to physicians during routine follow-up visits of pediatric oncology patients: Is it effective? Pediatric Blood & Cancer., 58(5), 766–774. https://doi.org/10.1002/pbc.23158
Haverman, L., Van Rossum, M. A., Van Veenendaal, M., van den Berg, J. M., Dolman, K. M., Swart, J., et al. (2013). Effectiveness of a web-based application to monitor health-related quality of life. Pediatrics, 131(2), 533–543. https://doi.org/10.1542/peds.2012-0958
Haverman, L., van Oers, H. A., Limperg, P. F., Hijmans, C. T., Schepers, S. A., Sint Nicolaas, S. M., et al. (2014). Implementation of electronic Patient Reported Outcomes in pediatric daily clinical practice: The KLIK experience. Clinical Practice Pediatric Psychology, 2(1), 50–67. https://doi.org/10.1037/cpp0000043
Schepers, S. A., Sint Nicolaas, S. M., Haverman, L., Wensing, M., Schouten van Meeteren, A. Y., Veening, M. A., et al. (2017). Real-world implementation of electronic patient-reported outcomes in outpatient pediatric cancer care. Psycho-Oncology, 26(7), 951–959.
Teela, L., van Muilekom, M. M., Kooij, L. H., Gathier, A. W., van Goudoever, J. B., Grootenhuis, M. A., et al. (2020). Clinicians’ perspective on the implemented KLIK PROM portal in clinical practice. Quality of Life Research. https://doi.org/10.1007/s11136-020-02522-5
van Oers, H. A., Teela, L., Schepers, S. A., Grootenhuis, M. A., & Haverman, L. (2020). A retrospective assessment of the KLIK PROM portal implementation using the Consolidated Framework for Implementation Research (CFIR). Quality of Life Research. https://doi.org/10.1007/s11136-020-02586-3
Jørgensen, K., & Rendtorff, J. D. (2018). Patient participation in mental health care—Perspectives of healthcare professionals: An integrative review. Scandinavian Journal of Caring Sciences, 32(2), 490–501.
Vahdat, S., Hamzehgardeshi, L., Hessam, S., & Hamzehgardeshi, Z. (2014). Patient involvement in health care decision making: A review. Iranian Red Crescent Medical Journal. https://doi.org/10.5812/ircmj.12454
Domecq, J. P., Prutsky, G., Elraiyah, T., Wang, Z., Nabhan, M., Shippee, N., et al. (2014). Patient engagement in research: A systematic review. BMC Health Services Research, 14(1), 1–9.
Jeremic, V., Sénécal, K., Borry, P., Chokoshvili, D., & Vears, D. F. (2016). Participation of children in medical decision-making: Challenges and potential solutions. Journal of Bioethical Inquiry, 13(4), 525–534.
Wiering, B., de Boer, D., & Delnoij, D. (2017). Patient involvement in the development of patient-reported outcome measures: A scoping review. Health Expectations, 20(1), 11–23.
Fischer, K. I., De Faoite, D., & Rose, M. (2020). Patient-reported outcomes feedback report for knee arthroplasty patients should present selective information in a simple design-findings of a qualitative study. Journal of Patient-Reported Outcomes, 4(1), 6.
Staniszewska, S., Haywood, K. L., Brett, J., & Tutton, L. (2012). Patient and public involvement in patient-reported outcome measures. The Patient-Patient-Centered Outcomes Research, 5(2), 79–87.
Mejdahl, C., Nielsen, B. K., Hjøllund, N. H., & Lomborg, K. (2016). Use of patient-reported outcomes in outpatient settings as a means of patient involvement and self-management support—A qualitative study of the patient perspective. European Journal for Person Centered Healthcare, 4(2), 359–367.
Mejdahl, C. T., Schougaard, L. M. V., Hjollund, N. H., Riiskjær, E., Thorne, S., & Lomborg, K. (2018). PRO-based follow-up as a means of self-management support—An interpretive description of the patient perspective. Journal of Patient-Reported Outcomes, 2(1), 38.
Thestrup Hansen, S., Kjerholt, M., Friis Christensen, S., Brodersen, J., & Hølge-Hazelton, B. (2020). “I am sure that they use my PROM data for something important”. A qualitative study about patients’ experiences from a hematologic outpatient clinic. Cancer Nursing, 43(5), E273–E282. https://doi.org/10.1097/ncc.0000000000000738
Lapin, B., Udeh, B., Bautista, J. F., & Katzan, I. L. (2018). Patient experience with patient-reported outcome measures in neurologic practice. Neurology, 91(12), e1135–e1151. https://doi.org/10.1212/wnl.0000000000006198
Lapin, B. R., Honomichl, R., Thompson, N., Rose, S., Abelson, A., Deal, C., et al. (2021). Patient-reported experience with patient-reported outcome measures in adult patients seen in rheumatology clinics. Quality of Life Research, 30(4), 1073–1082. https://doi.org/10.1007/s11136-020-02692-2
Snyder, C. F., Blackford, A. L., Wolff, A. C., Carducci, M. A., Herman, J. M., Wu, A. W., et al. (2013). Feasibility and value of PatientViewpoint: A web system for patient-reported outcomes assessment in clinical practice. Psycho-Oncology., 22(4), 895–901.
Basch, E., Artz, D., Dulko, D., Scher, K., Sabbatini, P., Hensley, M., et al. (2005). Patient online self-reporting of toxicity symptoms during chemotherapy. Journal of Clinical Oncology, 23(15), 3552–3561.
Carlsen, B., & Glenton, C. (2011). What about N? A methodological study of sample-size reporting in focus group studies. BMC Medical Research Methodology, 11(1), 26.
Temme, B. B. A. (1999). Leidraad groepsbijeenkomsten. B&A Groep.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Terwee, C., Roorda, L., De Vet, H., Dekker, J., Westhovens, R., Van Leeuwen, J., et al. (2014). Dutch-Flemish translation of 17 item banks from the patient-reported outcomes measurement information system (PROMIS). Quality of Life Research, 23(6), 1733–1741.
Gamper, E.-M., Martini, C., Petersen, M. A., Virgolini, I., Holzner, B., & Giesinger, J. M. (2019). Do patients consider computer-adaptive measures more appropriate than static questionnaires? Journal of Patient-Reported Outcomes, 3(1), 7.
Cella, D., Yount, S., Rothrock, N., Gershon, R., Cook, K., Reeve, B., et al. (2007). The Patient-Reported Outcomes Measurement Information System (PROMIS): Progress of an NIH Roadmap cooperative group during its first two years. Medical Care., 45(5 Suppl 1), S3–S11. https://doi.org/10.1097/01.mlr.0000258615.42478.55
Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., Yount, S., et al. (2010). The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology, 63(11), 1179–1194. https://doi.org/10.1016/j.jclinepi.2010.04.011
Haverman, L., Grootenhuis, M. A., Raat, H., van Rossum, M. A., van Dulmen-den, B. E., Hoppenbrouwers, K., et al. (2016). Dutch-Flemish translation of nine pediatric item banks from the Patient-Reported Outcomes Measurement Information System (PROMIS)®. Quality of Life Research, 25(3), 761–765. https://doi.org/10.1007/s11136-015-0966-y
Haverman, L., van Oers, H. A., van Muilekom, M. M., & Grootenhuis, M. A. (2019). Options for the interpretation of and recommendations for acting on different PROMs in daily clinical practice using KLIK. Medical Care, 57(Suppl 5), S52–S58. https://doi.org/10.1097/MLR.0000000000001061
Luijten, M. A. J., van Litsenburg, R. R. L., Terwee, C. B., Grootenhuis, M. A., & Haverman, L. (2021). Psychometric properties of the Patient-Reported Outcomes Measurement Information System (PROMIS®) pediatric item bank Peer Relationships in the general Dutch population. Quality Life Research. https://doi.org/10.1007/s11136-021-02781-w
Cella, D., Gershon, R., Lai, J. S., & Choi, S. (2007). The future of outcomes measurement: Item banking, tailored short-forms, and computerized adaptive assessment. Quality of Life Research, 16(Suppl 1), 133–141. https://doi.org/10.1007/s11136-007-9204-6
Santana, M. J., Haverman, L., Absolom, K., Takeuchi, E., Feeny, D., Grootenhuis, M., et al. (2015). Training clinicians in how to use patient-reported outcome measures in routine clinical practice. Quality of Life Research, 24(7), 1707–1718.
Schepers, S. A., Engelen, V. E., Haverman, L., Caron, H. N., Hoogerbrugge, P. M., Kaspers, G. J., et al. (2014). Patient reported outcomes in pediatric oncology practice: Suggestions for future usage by parents and pediatric oncologists. Pediatric Blood & Cancer, 61(9), 1707–1710. https://doi.org/10.1002/pbc.25034
We would like to acknowledge the Dutch National Health Care Institute for their funding. Additionally, we would like to thank all patients and parents of the KLIK panel for participating in the focus groups and/or completing the evaluation questionnaire. Finally, we would like to thank our colleagues for reviewing and Biomedia for sending out the questionnaire.
The data collection was supported by the Dutch National Health Care Institute.
Conflict of interest
The authors declare that they have no conflict of interest.
All procedures performed in this study were in accordance with the ethical standards of the international and/or national research committee (Medical Ethics Committee of the Amsterdam UMC—W18_023 # 18.034 and Medical Ethics Committee of the Amsterdam UMC—W19_272 # 19.324) and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Consent to participate
Informed consent was obtained from all individual participants included in the study.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The original online version of this article was revised: Figure 3 was updated.
About this article
Cite this article
van Muilekom, M.M., Teela, L., van Oers, H.A. et al. Patients’ and parents’ perspective on the implementation of Patient Reported Outcome Measures in pediatric clinical practice using the KLIK PROM portal. Qual Life Res 31, 241–254 (2022). https://doi.org/10.1007/s11136-021-02950-x