Background

Patient-reported outcomes (PROs) play an increasingly important role in health care [1]. In clinical practice, PROs are used to assess the health status and wellbeing of patients [2, 3]. In addition, PROs are used as screening and monitoring tools to support health care professionals in providing patient-centered care and facilitate communication between patients and health care professionals [1, 2, 4].

Electronic assessment of PROs is becoming established in research and clinical practice, not in the least because scores can be automatically calculated and provided in real-time to health care professionals [5, 6]. Moreover, mHealth technologies, referring to health care delivery via mobile devices [7], enable PRO assessments at home [8,9,10,11]. The increasing usage of smartphones in all age groups [12] and the high potential seen in mHealth solutions also for an older generation [13], support the new direction of PRO data collection using mobile devices. In this way, health care professionals are able to monitor patients, evaluate their health status and communicate with patients, outside of the clinic [10].

The provision of PRO data to patients, in the form of an automatically generated PRO feedback report, which can be directly accessed by patients after completion, is not common practice. Health care professionals frequently provide information on PRO results during the consultation [6, 14, 15]. In an outpatient setting, patients thereby have delayed access to information on their PRO results, if PROs are collected off-site. However, timely access to PRO scores enables patients to reflect on their progress and promotes active management of their health situation [3].

In orthopedics, the advancement of surgical techniques, as well as cost pressures in clinics, has led to a decrease in the length of hospital stays [16, 17]. The majority of the recovery time after knee replacement takes place in an outpatient setting, which requires a substantial degree of personal responsibility and active disease management by the patient [18]. One way of supporting orthopedic patients to actively manage their recovery phase after surgery is through early health education and training as well as access to health information such as PRO results by providing mHealth solutions [19,20,21,22].

A newly developed education app for knee replacement patients was introduced to support them during their treatment. The app includes information about the post-surgery recovery phase, such as advice on physical activity after surgery, medication and healthy diet. In addition, the PROMIS® measures of Physical Function and Pain Interference are included in the app. Patients are invited to answer these measures at predefined measurement time points. Currently, the PRO scores are only shown in the health care professionals’ dashboard. This information is used to track the recovery phase of individual patients and patient groups. Integration of a PRO feedback report into the app would allow patients to access and monitor their PRO results as well. This information could facilitate patient engagement, improve communication with health care professionals as the PRO feedback report could support patients to address health issues and concerns, based on PRO scores [23].

An automatically generated PRO feedback report for patients using the app is possible, and enables patients to access their PRO results immediately via smartphone. This raises the questions of what kind of information is of value to patients and how a report should look like in order to be easy to understand and evaluate.

Methods

Aim

The aim of the study is to develop a PRO feedback report for mobile devices, which can be integrated into the newly developed patient education app for patients after knee arthroplasty. This PRO feedback report should be comprehensible and provide valuable information for adult patients after knee arthroplasty, using the PROMIS® health domains of Physical Function and Pain Interference.

Development process of a PRO feedback report for patients

In a first step, we conducted a literature review to examine examples and results of PRO feedback reports that provides the basis for the subsequent development steps. The focus of the literature review was to identify publications describing any kind of PRO feedback report for health care professionals or patients, in particular those addressing issues of how to present PRO data. The literature search was carried out in Medline combining search components for PROs and feedback reports in PubMed. For the search component ‘PRO’, we used the Mesh terms ‘Patient Outcome Assessment’, ‘Patient Reported Outcome Measures’ and ‘Quality of Life’ and combined these by use of the Boolean operator ‘OR’. For the search component ‘feedback report’, we used the Mesh terms ‘Electronic Health Records’, Health Smart Cards’ and ‘Patient Portals’ and the entry terms ‘patient feedback report’ and ‘feedback report’ in the title and abstract and combined these by use of the Boolean operator ‘OR’. Both search components were combined by use of the Boolean operator ‘AND’ and filtered by language (German and English) and by studies concerning human subjects. We screened title and abstracts of the identified publications and reviewed full text of those meeting our inclusion criteria. We also screened general scientific literature on how to present data. In addition, members of the special interest group ‘Quality of Life in Clinical Practice’ of the International Society of Quality of Life Research (ISOQOL) were approached regarding any (personal) experience, articles, guidelines etc. on how to feedback PROs to patients.

In a second step, we developed two preliminary versions of a feedback report (a graphical display and a text-based version) relating to the PROMIS® domains of Physical Function and Pain Interference. Our research group included three experts experienced in quality of life and PRO research with an additional clinical background (a psychologist, a physician specialized in internal medicine and psychosomatic medicine and a nurse). As part of the development, we discussed what information to include in the reports based on the results of the literature review and previous experience with PRO feedback reports for health care professionals, followed by initial drafts and two revision rounds.

In a third step, the two preliminary report versions were discussed with patients in terms of the value of the information, comprehension and interpretation of the report using qualitative methods (semi-structured interviews including cognitive debriefing techniques). The methodological approaches are explained in more detail later.

Finally, we derived reasonable modifications for the PRO feedback report based on the results of the interviews and revised the PRO feedback report versions for patients according to the results of the interviews with patients. We discussed these in light of results of the literature search. An overview of the development process is provided in Fig. 1.

Fig. 1
figure 1

PRO feedback report development process – flow chart

Full size image

Patient interviews

Orthopedic (primary knee arthroplasty) patients at a university hospital in Germany were invited to participate in our study (convenience sampling) 1 to 2 days after surgery. The sample size was not predetermined, as we were aiming to reach data and thematic saturation [24]. We therefore continued recruitment until no additional codes or categories emerged from the interviews (thematic/data saturation).

Patients were eligible to participate if they were over 18 years, had sufficient knowledge of German (spoken and written), were able to understand the purpose of the study and gave written informed consent. Patients with cognitive limitations, or insufficient knowledge of German were excluded from the study. Ethical approval was obtained from the ethics committee of the university hospital.

The first author conducted all semi-structured interviews with patients according to the interview guideline during the hospital stay. Patients received printed copies of both report versions for both health domains in random order, as the patient app does not currently include any feedback report for patients. They were initially invited to look at the paper copies of the feedback report on their own. Cognitive debriefing methods for probing such as think-aloud and comprehension were used [25]. To assess comprehension of the feedback report drafts, patients were engaged to explain what they see, how they interpret the different elements, and what information they get out of the report. If an element of the report was not addressed by a patient, we explicitly pointed it out. Further, patients were asked which report style they prefer and if a feedback report would be of value and why.

Interviews were tape recorded and transcribed verbatim. All interviews were analyzed using a content analysis approach. Based on the guideline developed for the interviews, we pre-specified a category structure (Table 4) that is ‘general value of the report’, ‘favored report style’ and ‘comprehension’ and ‘informative content’ of both report styles. Additional categories were created, if necessary. All interview transcripts were read multiple times and codes were extracted and assigned to one of the specified categories. We subsequently summarized and discussed the pre-defined and created categories and included codes in detail. The interview analysis was performed using MAXQDA10 software for Windows.

Results

Development process of a PRO feedback report for patients

Review of the literature revealed only few studies focusing on PRO feedback reports in general, and on how to report PRO scores in particular. Results from previous studies showed that patients prefer short and condensed information [26] using simple language [27]. An efficient way to present longitudinal PRO scores is by use of graphs such as bar charts or line graphs [28,29,30,31,32,33,34,35,36,37,38,39], alternatively PRO results could be reported by use of tables [40, 41] or textual reports [31, 39]. A text-based report is the least preferred option for patients [31] but is less susceptible to misinterpretation [39]. For graphs, meaningful labelling of axes is recommended. For example, descriptive labeling, and harmonization of directionality, where higher indicates better facilitates interpretation of PRO scores [35,36,37, 41, 42]. Further, highlighting results of importance helps to focus attention and support the interpretation of results [30, 35, 36]. Color-shading, threshold lines or circles around important scores are suggested to increase readability and interpretability of graphical displays [14, 15, 29, 35, 36]. Additional information such as comparison to a reference population could be of value for some patients, but might be critical to others [34, 41, 43]. An overview of the results of the literature review is provided in Table 1.

Table 1 PRO feedback reports - literature overview
Full size table

In the initial development phase, we decided to provide feedback reports on a domain level (PROMIS® health domains Physical Function and Pain Interference), rather than a combined report, to increase readability on mobile devices, which was in concordance with the results of the literature review. We were aiming to include the feedback reports in an existing patient app that already includes a message function. Thus, we decided to develop two different report styles, a text-based report, which could be sent like using the message function, and a graphical display, which would be shown after the PROM completion. The text-based report was considered a pragmatic solution and could be implemented without additional effort, whereas, reprogramming would be needed to incorporate the graphical display. Our research team determined the following information to include in the PRO feedback report: the domain T-score, reference to the norm population, score thresholds and the predicted disease trajectory, as well as the opportunity to share information with health care professionals by use of the text message function or by a push button.

The text-based version included general information regarding the assessment, estimation of the current individual level of symptoms or function (including examples of limitations), and reference to the norm population. Patients were informed to contact their health care team in case of unexpected PRO changes or related questions. We used the thresholds provided by the PROMIS® Health Organization [49] to categorize value ranges. We created four categories for the Pain Interference health domain (group 1: T-score < 55; group 2: T-score of 55–60; group 3: T-score of 61–70; group 4: T-score > 70) and five categories for Physical Function (group 1: T-score < =30; group 2: T-score of 31–40; group 3: T-score of 41–45; group 4: T-score of 45–65; group 5: T-score > 65). Text messages were adapted according to the respective category. Patients scoring within the range of a category would receive the corresponding text message.

The graphical display included information on the individual T-score, the reference to the norm population, score thresholds and information on the predicted disease trajectory. The information was included on a line graph, where scores are plotted over time. On the x-axis, the measurement date as well as the time passed since the knee arthroplasty are displayed. We included two y-axis. The right-hand side y-axis only included the numerical scale. Descriptive labels were added to the left-hand side y-axis, referring to norm population scores. In addition, we added a rainbow-colored background from red (bottom) to green (top) to the line chart to visualize the grading of the individual scores. Whereas higher scores of Physical Function are better outcomes, higher scores of Pain Interference are worse outcomes. In order to account for this, we reversed the scale on the right y-axis for Pain Interference to harmonize the direction of the line graphs, i.e. the higher the better. We overlaid the line graph with an additional shaded area indicating the projected trajectory of the health domain. A button to share the information with health care professionals was included. For Pain Interference, we added information on the relative frequency of pain medication according to measurement points and percentage of change in pain medication in reference to the previous value. For Physical Function, we added information on immune activity including a rating of inflammatory markers (low, medium or high). The preliminary versions of the PRO feedback reports for the domain Pain Interference are displayed in Fig. 2 and Table 2.

Fig. 2
figure 2

Preliminary version of the graphical PRO feedback reports for the health domain of Pain Interference

Full size image
Table 2 Preliminary version of the text-based PRO feedback report for the health domain of Pain Interference according to grouped T-score categories
Full size table

Patient interviews

We invited 15 adult patients, who have underwent knee arthroplasty to participate in our study between September 2018 and January 2019. Seven patients who declined to participate said they were not interested in the study or were already participating in another research study. Non-responders were predominantly male (n = 6). We interviewed eight patients after knee arthroplasty, five women (62.5%) and three men (37.5%) aged between 60 and 80. We continued interviews until generated data was repetitive and no new codes or categories emerged during the analyses. This was the case after five interviews. The interviews took 24 min and 33 s on average, ranging from 19:59 to 28:38 min. Sociodemographic details of the study participants are provided in Table 3.

Table 3 Sociodemographic details (n = 8)
Full size table

To achieve the aim of the study and the purpose of the interviews, we predefined four categories and four sub-categories, which were general value of the report, favored feedback report style, graphical display, including the sub-categories comprehension and informative content, and text-based report including the sub-categories comprehension and informative content. During the coding process, the additional category aspects related to the individual person or situation emerged.

The general value of the reports was rated as moderate. While all patients thought such a report would be interesting and they would look at it, only one patient saw personal gain from using the report as a communication tool with health care professionals and as a self-monitoring tool. One participant stated that the majority of people would not understand what this report is about. Another participant stated that the idea of such a feedback report is good, but probably more suited to a younger generation. Another aspect mentioned by one participant was the course of treatment after knee surgery in Germany. This includes an extensive rehabilitation phase in a specialized clinic with daily treatments and regular examinations. The participant questioned the benefit of such a report given the continuous monitoring of the recovery process in Germany.

Participants had different opinions regarding the favored feedback report style. Three participants were in favor of the graphical display and two participants were undecided but tended toward the text-based version. Participants in favor of the graphical display argued that it is easy and quick to get the relevant information from the line graph. Those undecided stated that the graphical display looks more appealing, but the text-based version is easier to understand and most people are used to read short text messages.

The text-based report was easy to understand and none of the patients had questions regarding comprehension. Participants assessed the report as informative, but thought that some information was repetitive. Issues regarding the reference to the norm population were raised, as the percentage of people reporting similar symptoms or functional limitations would not be relevant. Difficulties in assigning a meaning to the number was also raised. One patient was even put off by it, feeling it would put certain pressure on someone to belong to the majority. Overall, the text-based version was considered to be too long, Participants therefore suggested shortening the text.

The graphical display of the reports was perceived to be complex due to the inclusion of a lot of information. Participants were concerned that this format may not be appropriate for an older generation. In terms of comprehension, all participants correctly understood the line graph and were able to interpret the scores. However, some patients needed some initial guidance on how to read a line graph. The rainbow-colored background was understood by all participants. With one exception, participants automatically attached value to the different shades of color. However, the shaded area indicating the projected trajectory of the health domains was not clear to patients. One participant did not ascribe any value to this area. Others understood this area as the range of possible scores, or as a way to draw attention to the area of the respective scores. Just two respondents interpreted the improvement over time as a prognosis for recovery after surgery. Only one participant understood the additional information regarding pain medication and related the information to the Pain Interference T-score. The additional information on immune activity was deemed to be too technical. Participants guessed that it might be related to the immune system, but could not explain what it meant, or relate this information to the Physical Function health domain. The function to share the information with health care professionals was clear to all patients.

During the interviews, five participants emphasized the relevance of aspects related to the individual person or situation. It was thought that a comparison to the norm population would not be of particular interest to patients after knee arthroplasty, as the individual before and after comparison of physical function or level of pain is of particular importance. Furthermore, reference to the knee replacement would help to relate the report to the individual patient’s situation. Statements like the activities which one might not be able to do due to the limitations of physical function might not reflect the individual’s activities. The report therefore loses the personal touch. One patient mentioned that the measurement time points should be flexible so that the patient can decide on the frequency of assessments.

An overview of the categories including the number of codes per category and examples of codes is presented in Table 4.

Table 4 Coding structure of the interviews, number of codes and coding example
Full size table

Revisions of PRO feedback reports for patients

As a result of the interview analysis, we modified the PRO feedback reports for patients to increase comprehension and the value of the reports. Minimal adjustments were done to the text-based version. This version was comprehensible to all patients, but the text was too long and some of the information was not seen as valuable to patients. We deleted the reference to the norm population. A sentence stating what the score measured as well as a sentence linking the assessment to the patient’s recent knee arthroplasty were added.

The graphical display required more extensive revisions to decrease complexity and thereby increase comprehension and value. We excluded the additional information regarding immune activity and pain medication as this information was misunderstood or not understood by patients and was not interpreted in reference to the measured health domain. The button to share the information with clinicians was enlarged to improve user-friendliness. In addition, we added a green tick to confirm that data were transmitted successfully. We reduced clutter within the line graph to increase readability and comprehension. The reference curve of the expected disease trajectory became an optional setting. The rainbow-colored background was reduced to a bar on the left-hand side y-axis. The data points plotted on the line graph include the scores, as well as the color corresponding to the rainbow-colored bar added to the y-axis. In addition, we decided to enlarge the most recent score to draw attention to it. The revised graphical display for the Pain Interference domain is depicted in Fig. 3.

Fig. 3
figure 3

Final graphical display of the PRO feedback reports for the health domain of Pain Interference

Full size image

Discussion

This study aimed to develop a PRO feedback report for adult knee arthroplasty patients that is easy to understand while providing valuable information. Results showed that the group of knee arthroplasty patients is heterogeneous in terms of age and experience with mobile devices and so are their demands on a PRO feedback report.

Based on the patient interviews, it was not possible to appoint a preferred report style. In previous studies, patients preferred graphical displays over a text-based report [31, 39]. Patients reported that the graphical display looked more appealing but the text-based feedback report version was easy to understand, even if partly too lengthy. Corresponding with our finding, Fried and colleagues [26] reported that feedback reports for patients should contain short and useful information instead of lengthy explanations.

Presenting data over time allows users to evaluate their individual progress and to compare current and previous scores. Patients valued the line graph presenting scores over time as important information, something which is not provided in our text-based version. This finding corresponds to the results reported by Brundage et al. [30] that patients preferred the depiction of multiple time points.

Interestingly, participants considered the reference to the norm population in the text-based version to not be of value and hence not relevant. However, they viewed the labels added to the y-axis and the color-coding within the graphical display positively. This implicit reference to the norm population supports the evaluation of individual domain scores. Similar findings were reported in previous studies that the inclusion of information referring to a reference group, such as labels of axis or integration of thresholds or a norm line, is helpful in evaluating the results [15, 29, 33, 36].

During the interviews, initial guidance on how to read the line graph was helpful to patients, and they suggested that some guidance should be given to all users before receiving the feedback report. Baldwin and colleagues [27] outlined that information should be provided in a simple way, including a simple interface design. This means that if additional guidance is required, the interface design might be too complex. However, a brief introduction to the PRO feedback report, including the functionality of the graphical display, could be implemented in the mobile app. The user could refer back to this information when needed. This training on how to use and interpret the feedback report could help to increase the utilization of the report among users [50].

The graphical display of the feedback report included additional information that overwhelmed participants. Considering the conceptual framework ‘mHealth for older users’ described by Wildenbos et al. [51], vision problems, cognitive restriction as well as attitudes and health-technology literacy might have influenced the perceived overload of the graphical display. Further, the use of medical terms instead of layman’s terms and the presentation of the clinical information (pain medication and immune activity) was difficult for participants to understand and prone to misinterpretation. This information was removed from the report to ensure that appropriate information is provided according to the health literacy level of patients using simple language, crucial aspects for a user-friendly interface [27, 51].

Another aspect study participants addressed during the interviews was the penetration of smartphone use within this specific target group. Only a few study participants had a smartphone and only one patient stated that he uses his smartphone for purposes beyond communication (phone calls and messages). When designing PRO feedback for mobile app use, two major challenges have to be considered. Firstly, the penetration of smartphone use in a specific age group has to be evaluated in a national context [52, 53]. Secondly, experience and user behavior with mobile devices is closely linked to the health-technology literacy level of the target group [51, 54, 55]. The field of mobile applications offers technological opportunities such as customizations and user guidance. In this way, it is possible to adjust the default settings of the graphical design of the report so that the interested and experienced user can fade-in additional information, but the default design does not overwhelm the inexperienced user [33].

Limitations

This study has some limitations that have to be considered with regard to the study results. There were some limitations to the study design. We did not include patient research partners in the study and discussed only the two developed feedback report styles with patients without testing alternative design formats including less or more information. The information included in the feedback report and the design would have been different by including patient research partners within the study. Furthermore, there would have been different formats to present the information, especially for the graphical display of the report. Thus, we are not able to make any statement in terms of the optimal design of PRO feedback reports for patients. However, the study’s goal was to develop a feedback report for mobile devices that provides valuable information to patients after knee arthroplasty, while being easy to understand and to evaluate. We have to test the developed feedback report to evaluate functioning and the practical value of sharing such PRO information with patients in future studies. A feasibility study to investigate how frequently patients look at the report and how the information facilitate communication between patients and health care professionals during the recovery period is needed.

We initially planned to conduct focus groups instead of semi-structured interviews to induce discussions among participants. We invited patients to take part in a focus group, but all patients preferred an individual interview. The coding was conducted by one researcher only. To increase reliability, all codes were discussed with a second researcher and if necessary, codes were removed from a category or assigned to another.

In addition, we were faced with a high non-responder rate in our study and recruited a small number of participants. However, we consider the number of patients to be acceptable given the narrow scope of the study and the saturation of data after five interviews. Data generated in the last interviews were repetitive and no new codes or categories emerged, which lead to the assumption of data and thematic saturation [24]. Nevertheless, the results of this study have to be interpreted cautiously. The findings are limited to the specific setting of the study and cannot be generalized.

Conclusion

The development of a PRO feedback report for knee arthroplasty patients for mobile app use is challenging due to the heterogeneous group of patients. The range of health-technology literacy levels within the target group has to be considered when developing an easy to understand and informative PRO feedback report for patients. The amount of information provided to patients has to be balanced to minimize the complexity of the report while maximizing the value for patients. A graphical display including short explanatory texts seems to be the most promising approach as a simple default version can be displayed, while additional information can be set as supplementary information for interested patients.