Background

Health care services are increasingly challenged to sustain high quality of care while facing increasing demand and financial shortcuts. Integrating the patients’ perspective in evaluating quality of health services has been urged to ensure that the patients’ judgment of health outcomes is considered in improving the quality of services [1]. Patient reported outcome measures (PROMs) are measures that provide data directly reported by the patient or the patient proxy [2] and thus reflect the patient’s perspective. PROMs can be used to evaluate the effectiveness of care, as well as safety; both aspects are considered as quality characteristics [3]. Despite the potential value of PROMs, the evidence that PROMs are included in routine clinical practice [4, 5] and quality management [6] is still scarce. While current work in health services research has focused predominantly on measures of failure such as hospital readmission or mortality rates, PROMs are promising as measures of success, such as improvement in functioning [7].

There are different approaches to collect relevant health outcomes for the monitoring of quality in health services through PROMs. One approach is to collect PROMs in routine clinical practice and use the information primarily for shared clinical decision making [8]. Clinicians consider PROMs as valuable for shared decision-making, if the purpose and the process of data collection complements routine practice and is not disruptive. Thus, if the appropriate infrastructure is not put in place, clinicians are likely to refrain from using PROMs [9,10,11]. If the information is aggregated at service level, it could potentially also inform the monitoring of quality of health services.

Another approach – which is of key interest for the present study – is to collect PROMs within the entire health system and use the data to monitor quality of care across hospitals. Such approach has been considered in national quality registries and specific national monitoring initiatives [1]. International societies have started to publish recommendations on how to integrate PROMs into registries [12,13,14]. If the information is collected already within the entire health system, PROMs data would become ideally not only available for improving quality of care but also for clinical decision making. However, how these systems succeeded in improving quality of care, whether they are suited for informing clinical practice, which stakeholders actually utilize the information once it is available within the health system, etc. remains yet to be examined. Furthermore, there may also be differences in how such systems have been set up, e.g. PROMS have been an integral part of their development or PROMs have been introduced in existing information systems later on. Therefore, there is a need to systematically examine existing systems to guide the future development of such systems, as well as the implementation in other clinical settings or countries.

The aim of this study was to examine supporting and hindering factors relevant to integrating patient-reported outcome measures (PROMs) in selected health information systems (HIS) tailored toward improving quality of care across the entire health system. To respond to this aim, the NHS PROMs Programme and the Swedish Hip Quality Register (SHPR) and Swedish Knee Arthroplasty Register (SKAR) were used as examples. These examples were chosen, since both aim to improve quality of care for the same clinical population by using PROMs and have received international recognition as evident through international peer-reviewed publications yet differ in their development. The NHS PROMs Programme is a fairly recent attempt to integrate PROMs on a national scale and is targeted toward four clinical populations. The Swedish Quality Registers often serve as reference systems given their extensive experience in running such information systems [15]. The Hip and Knee Arthroplasty Register were chosen for comparative reasons with the NHS PROMs Programme. Though the systems pursue similar goals and have the same target population, they are different in other aspects as outlined in Table 1. The NHS PROMs programme, the SHPR and SKAR are considered in this study as health information systems (HIS).

Table 1 Overview of selected information systems
Full size table

Methods

First, a systematic search and review [16] of internationally peer-reviewed literature was carried out to identify supporting and hindering factors relevant to integrating PROMs as outlined previously with regards to the selected information systems. The results of this review served as the foundation for building an interview guide. Second, expert interviews were performed to discuss and elaborate on the identified factors. The study design is outlined in Fig. 1.

Fig. 1
figure 1

Overview of study design

Full size image

To account for the complexity in the evaluation of HIS [17], the Health Information System Evaluation Framework developed by Yusof et al. [18] served as the frame of reference for data analysis in this study. This framework consists of four components, each with several dimensions; the organization (incl. Structure and environment), the human (incl. System use and user satisfaction), the technology (incl. System, information and service quality), and net benefits (incl. Positive and negative impact on potential end-users) and has been recommended and used for evaluation of health information systems [19,20,21].

Data collection

Part 1 systematic search and review

A systematic search and review is characterized by a systematic literature search and the subsequent critical review of its content to derive state of the art knowledge on a phenomenon. Such type of review incorporates any study type [16]. The literature search was conducted in December 2015, without time limit for publication dates, using the databases PubMed and SCOPUS. The search strategies were modified for each database to account for their particularities. The search terms are detailed in Additional file 1: Appendix 1. For the selection of articles, first abstracts were screened and then full-texts. The detailed inclusion and exclusion criteria for the abstract and full-text screening are outlined in Table 2. Once the full-texts were identified, general descriptive data about each paper (such as authors, year, and journal) as well as relevant information related to the components and dimensions of the HIS Evaluation Framework were extracted. For the latter, the primary interest was on how authors reflected and discussed their findings in light of the system. For instance, the time point of data collection (baseline and follow-up) was frequently stated in the Methods section of the papers. Any critical reflection in the papers upon these time points was of interest for the present study. The first author conducted the data extraction, the second author reviewed and challenged the extracted data on a regular basis; the data extraction sheet was then presented to an expert in PROMs (measurement) and HIS to gain his feedback on the plausibility of the information extracted and enhance credibility. No quality check of the identified studies was conducted since the aim was not to synthesize the previous results qualitative or quantitatively but to extract supporting and hindering factors of the HIS.

Table 2 In- and exclusion criteria for systematic search and review
Full size table

Part 2 expert interviews

In the context of this study people were considered as experts if they have been involved in research based on or related to the selected systems. To identify experts, research networks were identified using social network analysis [22]. In this study, a network reflects a group of people who have collaborated on work related to the system. The more a person has been involved in work related to the systems, the more central the person appears in the network. The networks were created based on the author information extracted from the literature identified in the first part of this study. The social network analysis for both countries respectively was conducted using the graph components of the networkx package of Python. Publications were iteratively analysed. For each individual publication all authors were inserted as nodes into the graph if they were not already existing in the graph. Each author pair (with n authors, one has n*(n-1)/2 pairs) represents an edge with weight 1 in the graph between the two corresponding authors. If an edge exists already, then the weight of this edge is incremented by 1. The networks, which resulted from this analysis, served as the foundation to identify experts with varying expertise.

From each identified network at least one person was invited for an interview. The intention was to interview 6 to 9 experts for each country. Twelve to 14 people were initially contacted. The guiding principles were: i) from each network authors with the highest number of publications would be contacted first; ii) the number of persons contacted from each network should be reflective of the actual network size. This approach allowed experts that are representative for the different research networks to be invited. Potential participants were contacted via e-mail with information about the study and an invitation to participate in a phone interview on a date of their preference. An information sheet was attached to the e-mail which outlined the purpose of this study, information about the data collection and analysis process, as well as data storage. Two reminders, the first after 4 weeks and the second after 8 weeks, were sent if no response was received.

The interview started with an open question about the goals of the specific system and whether the participant’s considered they had been achieved. Targeted questions were then asked based on the results of the literature review. This approach to developing an interview guide is consistent with qualitative research using directed content analysis [23] which was applied in this study (see Data analysis). A pilot interview was conducted with a person familiar with either of the two country examples. The interview guide was revised based on the feedback from the pilot interview. The final interview guide (see Additional file 2: Appendix 2 for the NHS PROMs Programme and Additional file 3: Appendix 3 for the SHPR and SKAR) was sent one day prior to the interview to each participant to remind him or her about the phone interview and provide some information about the main topics to be addressed. During the interview, the researcher elaborated on the background information relating to each aspect. For instance, response bias was identified as an aspect related to system quality and listed as a targeted question in the interview guide. During the interview, the researcher provided further information on the numbers and characteristics of non-responders, potential reasons for and threads due to this bias as described in the literature to stimulate the discussion. The interviews were tape-recorded after receiving verbal consent of participants at the beginning of the interview.

Data analysis

Part 1 systematic search and review

A directed content analysis was conducted [23]. Such type of analysis is suitable when some research about a topic exists already and can serve as the foundation for further research. In this study, the Health Information System Evaluation Framework served as the foundation to identify themes to be subsequently presented to experts in the interviews using directed content analysis. The components and dimensions of the Health Information System Evaluation Framework deployed in this study served as the guidance for the initial coding. The analysis of the literature was performed in a first step for the two country examples separately, and in a second step across examples. The identified papers were reviewed and relevant paragraphs, in particular from the Discussion sections, were assigned to the components and dimensions of the framework. Subsequently, each component and dimension along with the assigned paragraphs, were reviewed and more general themes identified. The identified themes informed the targeted questions for the interview.

Part 2 expert interviews

The interviews were transcribed verbatim and then coded using directed content analysis [23] as described already for Part 1. Each transcript was read several times and relevant paragraphs assigned to the components and dimensions of the HIS Evaluation Framework [18]. For any information that could not be assigned to the framework a new theme was created. Each component and dimension, along with the assigned paragraphs, were then reviewed to identify more general themes. Subsequently, all themes were checked against each other to avoid redundancies across themes and ensure that the final themes are mutually exclusive. Once the themes were identified, they were reviewed and assigned to either being a supporting, hindering or neutral factor. This assignment was done initially by the first author, then reviewed by the second author and revised accordingly. As in the literature review, the analysis was conducted first for the two country examples separately and then across examples.

Results

Part 1 systematic search and review

The electronic search resulted in 270 identified records for the NHS PROMs Programme and 335 records for the SHPR and SKAR together. After removing duplicates and applying the in- and exclusion criteria, 37 records for the NHS PROMs Programme and 26 records for the SHPR and SKAR were included in the further analysis. The details of the literature search flow are presented in Fig. 2.

Fig. 2
figure 2

Results of literature search

Full size image

For the NHS PROMs Programme, papers were mainly excluded because they described studies which informed the development of the NHS PROMs Programme or other initiatives to collect PROMs data and enhance quality through PROMs but not grounded within the NHS PROMs Programme. For the SHPR and SKAR, the main reasons for excluding papers were reference to other registries, no reference to PROMs, reference to either registry in the discussion but the registry was not a fundamental component of the study itself.

The directed content analysis revealed aspects of each component of the HIS Evaluation Framework. The most salient themes included aspects related to governance and funding, uptake of information by different stakeholders, the selection of PROMs and the linkage of data with other databases. These themes were then presented to experts within the scope of the interviews for further discussion. Throughout the interview, detailed findings from the literature review related to each theme and for the respective country example were presented by the researcher.

Part 2 expert interviews

Experts – NHS PROMs Programme. The social network analysis for the NHS PROMs Programme resulted in 5 networks as illustrated in the left part of Fig. 3. The networks had their main foci on economics, clinical practice, epidemiology and psychometry. Two networks reflected two research teams of which each published one paper. Out of the 13 people invited, 7 participated, 6 did not respond or declined. Experts included clinicians, economists, epidemiologists, and statisticians.

Fig. 3
figure 3

Results from Social Network Analysis

Full size image

Experts – SHPR and SKAR. The social network analysis for the SHPR and SKAR resulted in one network. The analysis revealed research collaborations across the two registries as represented in the right part of Fig. 3. Thus, in this case experts were invited not only based on the highest number of publications, but so that they were representative for the hip and knee registries. Out of 13 people who were invited for an interview, 2 attended a phone interview, one from SHPR and one from SKAR, and 1 expert from the SKAR responded via e-mail. Six did not respond and 4 declined or referred to others who already agreed to participate. All experts who participated in the interviews were clinicians by background; only one worked primarily as such at the time of the interview.

All interviews for both, the UK and SE system were conducted between end of February and April 2016 and lasted on average 51 min (min. 25 min – max. 1,5 h).

Interviews. The majority of the experts highlighted that the integration of the PROMs into the HIS and its success has to be considered in light of the goal for which the data collection was set up. The experts for the SHPR and SKAR highlighted that the goals of the registry, namely to improve quality of care, did not change through the integration of the PROMs. One expert of the SKAR highlighted that the integration of the PROMs allowed the registry to realize a bio-psych-social perspective in evaluating and monitoring outcomes which is important since

“We are operating people, not knees.” (SE_3_L111).

The experts with reference to the NHS PROMs Programme emphasized that the goal of the programme was broadly and vaguely defined whereby improving quality of care was one goal amongst others. Most experts highlighted that the program demonstrated that a nation-wide PROMs data collection is feasible but the achievement of other goals, such as improving quality of care, remain yet to be proven.

The factors related to each system which revealed from the analysis are detailed by components of the HIS Evaluation Framework in Tables 2, 3, 4, 5 and 6. For the two Swedish registries, it is explicitly stated if a statement was only provided for one registry. No inferences can be made that this does or does not apply to the other registry since no information on the topic was given by the experts of the other registry. Points that were controversially discussed in relation to each component are elaborated on in the subsequent sections. Only results are presented that revealed of relevance in the analysis across the country examples.

Table 3 Results of expert interviews related to the component Organization of the HIS Evaluation Framework
Full size table
Table 4 Results of expert interviews related to the component Human of the HIS Evaluation Framework
Full size table
Table 5 Results of expert interviews related to the component Technology of the HIS Evaluation Framework
Full size table
Table 6 Results of expert interviews related to the component Net benefits of the HIS Evaluation Framework
Full size table

Organization

The SHPR and SKAR built upon decades of experience in building up the infrastructure for the registries, and thus reported about a rather consolidated governance and infrastructure as reflected in Table 3, left column. In contrast, the experts related to the NHS PROMs Programme reflected more critically upon these themes since they considered the system to be in its infancy and the governance as decisive for the success and sustainability of the programme (Table 3, right column). For instance, experts suggested that more involvement from clinicians would have been beneficial from the very beginning on. However, some argued that it would have required more time resources to agree upon the data collection protocols back then since there was a lack of outcome measurement conducted in routine practice. Overall, a lack of capacity for quality management within the NHS was identified.

“It is the responsibility of the health care system in general to develop staff so that they can manage quality and we don’t have that. (…) There isn’t senior leadership in hospitals with responsibility and the skills and knowledge as to how to improve quality. So in a way what we got is increasingly sophisticated national data getting produced. But actually it is falling on stony ground because people don’t know because they haven’t been trained how to use it.” (UK_1_L207–213).

Human

The reporting and engagement with stakeholders revealed as important themes. Both are measures of the success of an information system according to the HIS Evaluation Framework. The detailed findings related to these themes are presented in Table 4. The standard way of reporting results of the NHS PROMs Programme is with funnel plots. Prior to preparing the plots, the data is adjusted for case-mix. Different opinions amongst experts revealed on whether the data is easy to analyse and interpret:

“There is enough technical guidance for someone reasonable capable to deal with PROMs data – it is pretty straight forward.” (UK_2_L252–253).

“I think the problem with this – as I see it – if the aim is to provide more data for patients and commissioners to increase choice and accountability, I think the reporting processes presently aren’t robust enough to allow that to happen. By that I mean the data that comes out is difficult for commissioners and patients to interpret, it is statistical analysis of it, and the requirements of the statistical analysis to ensure that the data is analysed is appropriately are probably too complex for commissioners and patients to understand.” (UK_4_L11–16).

This complexity and related lack of transparency impedes end-users to engage with the data:

“You can’t dig into it, you can’t ask questions of it. You have just take on face value of what they give you, which is a bit disappointing. Unless you can dig into it and understand the data, you cannot identify the area in which you need to improve to get better.” (UK_4_L325–327).

Technology

All participants, irrespective of the system, stressed that as a basic principle the questionnaires should be not too long to keep the burden on participants in an acceptable range:

“Simple data collected comprehensively is better than comprehensive data collected simply or poorly.” (SE_3_L140–141).

Experts from both country examples mentioned some alternative PROMs but in consideration of their length and popularity considered the current selection as appropriate. Next to the aspects which are shown in Table 5, the majority of experts challenged whether it is appropriate to apply existing population preferences to national PROMs data. In particular if such data is then used to inform the treatment of individual patients. For instance, for the EQ-5D existing population preferences are used to transform the scores into utility measure. Some experts argued that preferences specifically tailored to the purpose of the NHS PROMs Programme are needed. Such research efforts are ongoing. Others argued that the population preferences are informative for the individual patient, e.g. to know what the population feels about the trade-off between different dimensions covered by the EQ-5D, but should not be the basis for the final clinical decision making.

The NHS PROMs Programme includes a section where patients are asked about any complications in the context of the surgery. Experts were very critical upon these questions being patient-reported. Since differences in the interpretation of clinicians and patients in their judgment of complications are to be expected. While clinicians may introduce a systematic bias, e.g. by rating certain things always or never as complication, patients may overestimate complications. One expert stated that cross-linking the PROMs data with the Hospital Episode Statistics (HES) did not reveal much agreement. The HES contains data on complications as judged by clinicians.

The linking of PROMs data to other databases provided more of an added value to the SHPR and SKAR whereas it was rather essential for the NHS PROMs Programme to generate a reasonable data set. The aspects which revealed regarding linkage are listed in the bottom part of Table 5. Experts of the NHS PROMs Programme mentioned that some information is collected in various databases but with different modes. Examining how the data on one specific variable varies by administration mode is meaningful for the verification of the credibility of the data. Thus, data linkage does not only allow to extend the data but also to enhance its quality.

Net benefits

In addition to system-specific considerations listed in Table 6, one of the areas mentioned by all experts with controversial opinions is the benefit of PROMs for clinical management and decision making. To strengthen the usefulness of PROMs further for clinicians and patients, the SHPR is currently developing a decision support tool which will provide individualized information on a given patient’s risks and potential benefits of a surgery.

“The information will be based on what we have in the registry. It is like patients like you generally have a 60 % chance of improving in usual activities in the EQ-5D. Given your responses now, and given your age, gender, socio-economic background, you would have this and this chances of improvement or risk of suffering from complication.” (SE_2_L169-L170).

The Decision Support Tool will be available on the clinician’s portal; whether a patient portal would be created was still an open question at the time of the interview. Several experts of the NHS PROMs Programme mentioned that the integration of PROMs data into routine clinical practice for decision making would make another success of the programme. Nevertheless, other experts from both country examples challenged whether the PROMs used in these systems are appropriate for use in individual decision making since they were all developed for clinical trials to compare intervention groups against control groups. One expert from the NHS PROMs Programme stated that the data available now could serve to establish the baseline population and to develop trajectories of these patients to inform future practice.

If there are different kinds of patients, you know, you can stratify it by gender or age, and by clinician type and communicate that kind of information is very meaningful to patients who fit that profile. I think this is part of evidence-based medicine. (UK_6_L77–80).

The lack of such a baseline made another expert from the SKAR sceptical about the readiness of PROMs for use in clinical practice. The value of PROMs in managing access to care was also mentioned by experts from all systems. Most experts were sceptical since no evidence is available yet on the impact of such practices. Concerns were voiced that patients and clinicians may start gaming the system.

“Patients would find out that we are stratifying on a filter questionnaire they will all game the questionnaire.” (SE_3_L467–468).

“We don’t know the impact how it is going to change the way how people are managed, how it is going to change (…) For instance surgeons might just want to choose patients who are going to do well definitely so that it looks good at them, so that they have good outcomes if it is something they are going to be judged on. And it can also go to the hospital level where you can have patients who actually need help but will not necessarily benefit based on the OKS. They might be left out of the system to their own devices which may bring extra cost to the overall health care etc.” (UK_5_L175–191).

An important issue mentioned for the SHPR and the NHS PROMs Programme is the need to have immediate access to data once it is used for clinical decision making.

“The issue is how quickly the data can be made available for surgeons, and also for clinicians and hospitals, commissioners group and those from practice side who would like to have high quality-data on patients as soon as possible.” (UK_2_L96–98).

The cleaning and validating of the data does not take place real-time in either system and is essential for sustaining high data quality in the registry. The expert from the SHPR stressed that using the data for clinical decision making should not defeat the quality of the registry and its purpose, which is quality improvement and monitoring of outcomes.

Discussion

This study examined supporting and hindering factors of the NHS PROMs Programme and Swedish Hip and Knee Arthroplasty Registry with respect to the integration of PROMs within the entire health system. The country examples are representative for a system that integrates PROMs into an existing HIS (SHPR and SKAR), and a system that has been set up independently and specifically for collecting PROMs data (NHS PROMs Programme). Based on the experts’ opinions, these country examples were successful in their efforts to collect PROMs within the entire health system.

PROMs assist in improving the quality of care according to experts of this study. The findings highlight the role of PROMs to realize a bio-psycho-social perspective on health and may constitute one important step toward going beyond solely biomedical outcomes in improving quality of care. This is in line with the argument previously made that PROMs have the potential to stimulate the move beyond traditional outcomes of mortality and morbidity [7]. Nevertheless, it is also fair to say that based on the findings of this study the value of PROMs for improving quality of care remains yet to be established in the UK. One of the main concerns for using PROMs in monitoring quality of care is the lack of an established standard on what change in PROMs scores should be achieved when conducting a surgery for arthroplasty. Such knowledge is needed if PROMs are to be used for monitoring the quality of care since “quality is not represented by health status but by the extent to which improvement in health status that are possible are realized” ([24]; p 10). The existing data sets collected within these country examples could serve to establish evidence on what change can be expected.

Though all experts highlighted the value of using the PROMs data collected within these systems in routine clinical practice, this study sheds light on some important considerations. The combination of generic and health-condition specific instruments suggested previously [13] is also supported by the experts who participated in this study. This combination provides clinical face validity for treating patients with a particular health condition and enables comparisons across health conditions. However, experts challenged whether the data collected within the country examples is reliable for routine clinical decision-making, since the reliability of these instruments has been tested previously mainly on group level. Concerns were also mentioned with regard to the use of PROMs to guide access to care. Furthermore, as already highlighted by Boyce, Browne and Greenhalgh [9], the infrastructure for collecting and retrieving PROMs data revealed also in this study as a main source for success or failure for integrating PROMs into routine practice. The need for finding a balance between providing high data quality and making the data available instantly for routine use was identified as challenging. Future research may address some of these open questions on the use of the PROMs data collected within these systems in routine clinical practice.

The main difference between the country examples refers to the organization of the systems, in particular their governance. The NHS PROMs programme enacts a top-down approach whereas the Swedish registries a bottom-up approach. Various stakeholders, including commissioners, economists, patients, and surgeons were identified for the NHS PROMs programme. The variety of the stakeholders of the NHS PROMs Programme could be seen as strength, though experts in this study identified the need for providing reports that are better tailored to the needs of the different stakeholders. Having commitment of the government, in particular with respect to funding, was identified as supporting factor of the NHS PROMs Programme and a weakness of the Swedish registries. The top-down approach deployed in the NHS PROMs Programme resulted in a lack of ownership of the data by clinicians. In contrast, clinicians were the key stakeholders stated for the Swedish country examples. They were involved in finalizing reports. The limited involvement of clinicians in the NHS PROMs Programme was identified as a thread for its sustainability. Taking the identified supporting and hindering factors of both country examples into account, a participatory approach with involvement and commitment of stakeholders at the clinical and managerial level would be ideal. Such comprehensive approach would be also in line with knowledge on the successful implementation of innovative programs in health care [25].

The credibility of the findings of this study need to be considered in light of the methods deployed. Reviewing the literature systematically to inform the interview guide can be considered as a method triangulation to enhance the rigor of this study [26]. All experts stated that they consider the most relevant themes captured by the interview guide which was taken as confirmation of the comprehensiveness of the review. The selection of experts based on the social network analysis allowed identifying experts with various areas of expertise (see Table 4). Nevertheless, the basis for generating the networks was a pool of people who have published internationally any work in relation to any of the two systems. Thus, for future research it is recommended to include various stakeholders, including politicians, patients and service users of these systems. Noteworthy, this study only focused on two selected information systems with a focus on two specific clinical populations from two different countries. Similar systems do exist for other health conditions and in other countries. Thus, also additional information systems and their relevant stakeholders need to be included in future research.

The unequal number of experts related to the NHS PROMs Programme and the Swedish registries is reflective of the organizational structure of these systems. The SKAR and SHPR are organized around a register centre which is responsible for the continuous registry work, including clinical research. Expert affiliated with the register centre of the SKAR and SHPR participated in this study. Given their position in relation to the registers, it can be assumed that they have a comprehensive knowledge about the supporting and hindering factors of these registers.

Recommendations to overcome some of the weaknesses of phone interviews in comparison to face-to-face interviews were integrated into this study [27]. For instance, participants received a reminder of the time of the interview a day prior to the interview along with an overview of the interview guide. This approach allowed re-confirming the date and timing and getting the participants into the scope of the interview. Furthermore, the speakerphone/tape recording method, which was used in this study, was tested prior to the interviews [28].

Directed content analysis [23] foresees the use of theory as guidance in the initial coding. It has been argued that this method may introduce a strong bias into the results since it is easier to find evidence that is supportive rather than unsupportive of the theory. In this study, the HIS Evaluation Framework [18] served as the frame of reference to ensure that a comprehensive perspective on HISs is enacted and all components relevant for a HIS are taken into consideration. No theoretical guidance on how these components and dimensions are intended to interact were adopted from previous research in this study.

Conclusions

Both country examples succeeded in integrating PROMs into HIS on a national scale. The value of adding the patient perspective in quality improvement by integrating PROMs in HIS tailored toward this purpose was stressed by experts from both country examples. Though the systems examined in this study are different in their historical development and the political context in which they are embedded in, there was agreement amongst experts that any such system is valuable and needs to be kept simple and clear. The lack of an established standard on what change PROMs should be achieved by an intervention was considered as a limitation of the usefulness of the PROMs for monitoring quality of care. Whether the PROMs data collected within these systems can be used in routine clinical practice is considered a challenge in both countries. The findings of this study can inform the further development of the existing systems as well as the implementation of such systems in other health conditions or settings. For the development of similar HISs in the future, a participatory approach with involvement and commitment of stakeholders at the clinical and managerial level can be recommended based on the findings of this study.