Background

Capturing patient outcomes and experiences is critical to enabling ongoing review and improvement of healthcare services [1]. Patient experiences can improve health care services towards greater patient-centred care by understanding the nuanced interactions with a health service and its practitioners. Patient outcomes can be used to capture a person’s perception of their own health, such as their quality of life, daily functioning, symptoms, and other aspects of their health and well-being. Evaluating both outcomes and experiences of a patient’s health journey permits a holistic insight into healthcare services and ensures they are valued with a patient-centred focus.

Healthcare digitisation has provided an unprecedented opportunity to collect patient information in more recent years via an electronic medium in the form of electronic patient-reported outcome measures (ePROMs) and electronic patient-reported experience measures (ePREMs). ePROMs are digital questionnaires measuring patients’ views on their health status, symptoms, daily functioning, quality of life, and other characteristics of health and well-being [2]. ePREMs, on the other hand, are digital questionnaires measuring patients’ perceptions of their experiences while receiving care [2]. A 2019 study suggested ePROMs, when compared to traditional paper-based patient-reported outcome measures (PROMs), improve data quality, result in similar or faster completion times, lower costs, and facilitate clinical decision-making and symptom management [3]. However, disadvantages of digital delivery include privacy concerns, large initial financial investment and the ‘digital divide’ disadvantaging patients who are not digitally engaged [3]. Furthermore, there is limited information on the use of ePREMs, and the facilitators and barriers to implementation [4].

ePROMs/ePREMs allow for patient monitoring across all levels of the healthcare landscape, from individual clinician-patient ‘point-of-care’ interactions (micro level) to population surveillance and informing policy (macro level) [5]. Integration of patient-reported data with clinical and service delivery data can enable a holistic view of patients’ overall care journey, while concurrently enabling the monitoring of service performance to identify gaps and opportunities for enhancement, creating a positive feedback loop for improvement [5, 6]. Despite the apparent opportunities provided by digital technology, in practice the implementation and uptake of ePROMs, and especially ePREMs, remains sparse and inconsistent. Barriers to ePROM uptake have been reported, such as insufficient stakeholder engagement and training, a lack of interoperability with existing clinical information systems, and data management [7,8,9]. There is little known about whether the barriers and facilitators for the implementation of ePREMs and ePROM overlap.

To address these knowledge-to-practice gaps in the utilisation of ePROMs and ePREMs across healthcare settings, this study aimed to systematically review the literature to a identify facilitators and barriers to implementing both ePROMs and ePREMs in a healthcare setting, and map the identified facilitators and barriers to the Consolidated Framework of Implementation Research (CFIR) determinant framework. CFIR is a commonly used implementation science framework to facilitate the design, evaluation, and implementation of evidence-based interventions by systematically assessing potential facilitators and barriers for implementing an intervention at all levels of the healthcare system, including individual, organisational, and beyond [10,11,12,13]. In comparison to other frameworks, such as the theoretical domains framework, CFIR was chosen for its comprehensive examination of the various tiers of healthcare and the various implementation strategies necessary for success. The study also aimed to summarise the findings into an actionable checklist that defines best implementation practices.

Review question

What factors (facilitators and barriers) are currently identified in the literature as influencing the implementation of ePROMs and ePREMs in health care?

Inclusion criteria

The following eligibility criteria were developed to frame the review, based on a modification of the criteria used in a previous systematic literature review of PROM implementation facilitators and barriers [14].

Population

Studies including patients, clinicians, commissioners, or managers of health-related services. Commissioners could be representatives of either local or national agencies that finance health-related services (e.g., policymakers).

Phenomena of interest

Studies that investigated factors reported to influence the implementation of ePROMs and/or ePREMs.

Context

Studies investigating health-related services, irrespective of patient populations, providers, sectors of health care or country.

Articles that met all the following inclusion criteria were included in the review:

  1. 1.

    Include studies reporting facilitators and barriers (or factors) that influenced the implementation of ePROMs and/or ePREMs.

  2. 2.

    Have a digital delivery method that involves the sharing of information including data and content among mobile phones, computers, and tablets.

Articles were excluded if they were:

  1. 1.

    Protocol papers (no results reported).

  2. 2.

    Studies on the issues of implementing ePROMs and/or ePREMs in research contexts (e.g., clinical trials) rather than in a clinical context.

  3. 3.

    Development or usability studies and other pre-implementation research studies.

  4. 4.

    Reports indicating fewer than 75% of participants (e.g., clinicians undertaking surveys) actively use ePROMs and/or ePREMs.

Type of studies: This review considered quantitative, qualitative, and mixed-methods studies.

Methods

This systematic review was registered in PROSPERO (registration number: CRD42022295392) and follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for reporting systematic reviews [15]

Search strategy

The search strategy keywords included patient-reported outcome measures; patient-reported experience measures; implementation; and electronic, digital, and mHealth. The full search strategy is reported in Additional File 1.

Information sources

Study selection

An electronic data search of five electronic databases (PubMed, CINAHL, PsycINFO, Web of Science, and Scopus) was conducted between the database inception and the search date (24th of January 2022). Search results were exported to Covidence [16], and duplicate entries removed. Manual screening of the reference lists of all included studies was performed to identify articles not identified in the database search. Two reviewers (BG and JS) independently screened the study titles and abstracts for inclusion in the full-text review, and then reviewed the full-text articles for final inclusion in the systematic review. Disagreements regarding article eligibility were resolved through discussion between the two reviewers. Where agreement could not be reached, a third reviewer (MC) adjudicated as necessary.

Assessment of methodological quality

Two reviewers (BG and JS) utilised the Mixed-Methods Appraisal Tool (MMAT) to assess the methodological quality of the included studies [17]. The MMAT contains a checklist of five sets of questions, each corresponding to a specific study design category. Each question must be answered as either “yes,” “no,” or “cannot tell.”. According to Coates et al. [18], studies were considered of high quality when meeting 100% of the criteria, moderate quality when meeting 80–99% of the criteria, average quality when meeting 60–79% of the criteria, low quality when meeting 40–59% of the criteria, and very low quality when meeting < 39% of the criteria. Both authors determined consensus quality ratings for each study, from which the overall percentage score for each study was derived. Agreement between the two reviewers for the quality assessment was calculated as both percent agreement and first-order agreement coefficient, as defined by Gwet’s AC1 [19, 20], and computed using the irrCAC (version 1.0) package [20], in R software (version 4.0.3) [21].

Data extraction

A single reviewer (JS or BG) extracted data pertaining to the characteristics of the included studies, such as the country of origin, clinical setting and type of study. Studies were categorised based solely on the relevant data extracted for this review (e.g., some studies collected both quantitative and qualitative data but if only the qualitative data was extracted these studies were placed in the qualitative category).

Data transformation

A descriptive code-based synthesis of the results was performed using qualitative descriptive coding to identify and synthesise the relevant findings of the included studies. Two reviewers (BG, JS) independently extracted findings from the included studies stating factors relevant to the implementation of ePROMs/ePREMs (e.g., specific participant quotes, themes derived by the authors of the included study, discussions points and ‘qualitized’ descriptions of findings from quantitative results).

Data synthesis and integration

The two reviewers coded all extracted data into bottom-level codes. Using content analysis [22], each reviewer independently grouped their set of bottom-level codes into top-level code categories using an inductive approach. Through discussion, the two reviewers collaboratively created a final consensus set of code categories and mapped these to relevant CFIR constructs.

Many of the resulting code categories were bidirectional, so they could be considered facilitators or barriers to implementing ePROMs/ePREMs in healthcare settings, depending on their execution. The two reviewers classified code categories as facilitators if more than half of the codes in that category were deemed to be facilitators (and, similarly, for barriers).

Results

Study inclusion

Figure 1 shows the preferred reporting items for the systematic reviews and meta-analyses (PRISMA) [23]. The search strategy yielded 626 records, of which 187 full-text were assessed for eligibility and 24 studies met the inclusion criteria.

Fig. 1
figure 1

PRISMA flow diagram of study selections

Full size image

Methodological quality

All articles were assessed and assigned to a category under the MMAT [17]. As shown in Table 1, eight studies were rated as high quality [24,25,26,27,28,29,30], seven as moderate quality [7, 8, 31,32,33,34,35], five as average quality [9, 36,37,38,39], two as low quality [40], and two as very low quality [4, 41]. No study was excluded based on quality assessment. The percent agreement for the quality assessment (excluding the MMAT screening questions) between the two reviewers was 81%, with a Gwet’s AC1 of 0.79 (95%CI 0.72–0.86) indicating good agreement. Full MMAT scores for all included studies, are reported in Additional File 2.

Table 1 Characteristics of included studies
Full size table

Characteristics of the included studies

The full details of the study’s characteristics are presented in Table 1. All studies were published after 2012, and conducted in eight countries: Netherlands [8, 31, 33,34,35], Finland [42], Denmark [26], United Stated of America [7, 9, 24, 27, 28, 30, 32, 38, 40, 41, 43], Italy [4], Canada [25, 27, 36, 37], Germany [39], and the United Kingdom [29]. The majority of studies (22/24) investigated ePROMs in isolation, one study investigated only ePREMs [4], and one study investigated both ePROMs/ePREMs [25]. The most common clinical setting was oncology, with 29% (7/24) of the included studies [8, 33, 34, 36,37,38,39, 42], followed by general hospital settings (3/24, 12%) [4, 32, 35]. Study designs were primarily qualitative (13/24) or mixed methods (7/24), but also included four observational quantitative studies. Across both qualitative and mixed-methods study designs, qualitative data were collected through interviews in 12 studies [24,25,26, 28,29,30, 32,33,34, 38,39,40, 43], focus groups in six studies [7, 9, 25, 29, 32, 36], and open-ended survey questions in five studies [4, 8, 31, 35, 38]. In studies that included a quantitative component, quantitative data were collected via close-ended survey questions in 11 studies [8, 27, 31,32,33,34,35,36,37,38, 40], a workflow audit in only one study [40], and Q-sorting in only one study [32].

Findings of the review: facilitators and barriers to implementing ePROMs/ePREMs

A total of 96 code categories were generated, of which 60 were classified as facilitators and 36 as barriers. Code categories were subsequently mapped to 26 constructs across all five CFIR domains. Table 2 provides quotes for the five most prevalent categories (facilitators or barriers) mapped to each CFIR domain with the prevalence of each code category defined by the number of studies containing at least one code for each individual category. A full list of all identified categories mapped to individual CFIR constructs can be found in Additional File 3: Table S1.

Table 2 Example quotes for the top five most prevalent categories mapped to each CFIR domain (including ties)
Full size table

CFIR domain: intervention characteristics

Within the included studies, stakeholders noted several areas in which ePROMs and ePREMs offered a relative advantage over alternative methods for gathering patient data. These advantages include facilitating information extraction that might be overlooked or not uncovered in the consultation [8, 24,25,26,27, 29, 34, 35, 38, 43], the potential for earlier detection of issues [24, 25, 29, 33, 35], how readily ePROMs/ePREMs allow for monitoring changes [24, 25, 28,29,30,31,32,33,34,35,36, 43], and the ease with which comparisons can be made between patients and peers [33]. Further advantages of the implementation of ePROMs are the efficiency it allows in allocating finite hospital resources [26], and the reduced need for in-person consults [26].

Regarding the complexity of the intervention, barriers emerged when ePROMs were perceived by clinical staff as a time burden due to being delivered too often or being too lengthy or repetitive [8, 24, 27, 30, 33,34,35,36, 39, 43]. Additionally, several studies have identified ePROMs can flag too many concerns to be discussed within the limited clinic time available [24, 25, 28, 30, 34, 38, 43]. Despite this, several studies acknowledged these complexities could be minimised by selecting an ePROM/ePREM platform with user-friendly software and technology that automatically captures, summarises and displays patient data [4, 9, 28,29,30,31, 35, 38].

Design quality and packaging of ePROMs were prominent constructs identified in 17 studies. Several studies have noted the benefits of ePROMs in providing an insightful, easy to read patient summary or snapshot [9, 29, 32, 35, 36, 43], along with graphical visualisations to easily see trends [8, 28,29,30, 32, 35, 38, 39, 41, 43]. It was also considered important to ensure patients received ePROM results promptly after appointments [26]. On the other hand, a significant barrier emerged when it was difficult to visualise data or obtain a summary due to limited visualisation options [30, 34]. The inability to distinguish potentially abnormal results due to a lack of clearly defined threshold scores was considered a barrier [9, 24, 29, 32, 36, 43]. In contrast, ePROM systems which clearly identified results that exceeded threshold scores and supplied real-time notifications of those requiring urgent responses was considered to support implementation success [30, 35].

Facilitators of implementation also included adaptive technology to automatically trigger additional ePROMs, based upon earlier scores [30, 33, 35, 36], and systems that presented questions one at a time to patients, which was considered less overwhelming to patients than large numbers of paper forms [28, 35]. The automatic integration of ePROM results into electronic clinical notes [30, 38], as well as providing clinical staff with real-time access to ePROM completion status and results both prior to and during consults were highlighted as facilitators [30, 35]. Other facilitators included the provision of automatic reminders for clinicians to discuss ePROM results in consult time [28, 34], and having ePROMs available in different languages [28, 30, 35, 36]. In contrast, unreliable or unstable software and hardware was identified as a barrier [4, 8, 9, 29,30,31, 33, 35, 40, 41]. Additional barriers to implementation included ePROMs not being completed at a clinically meaningful timepoint [30, 31, 33,34,35, 41], and the prohibitive costs associated with implementing ePROMs/ePREMs [7, 25, 33, 37, 39].

CFIR domain: outer setting

Studies have provided mixed views on patient needs regarding the implementation of ePROMs and ePREMs. Some studies have acknowledged patient frustration due to the lack of feedback from clinicians regarding ePROM results [26, 28, 35, 43]. The completion of ePROMs/ePREMs are considered difficult for patients with low language or computer literacy [24,25,26,27,28, 30, 31, 40, 41], and for patients with physical or cognitive impairments [24,25,26,27,28, 30, 35, 40]. Alternatively, ePROMs allowed patients to better communicate and prioritise their concerns with clinic visits [8, 24, 26, 28, 30, 31, 33,34,35,36, 38, 41], while amplifying the patients’ voice, thereby enhancing patient-centred care and shared decision-making [24,25,26, 28, 29, 31,32,33,34,35,36, 39, 43]. Furthermore, increased patient motivation to complete ePROMs was noted when patients felt they were contributing to ongoing research [34], while also leading to improved patient satisfaction and experiences [4, 28, 37].

Some studies have identified implementation barriers around patients’ level of comfort with, and access to digital technologies [26, 28, 30, 35]. Patients’ lack of awareness with regards to why they were being asked to complete the ePROMs was considered another barrier [24, 27, 30, 33, 34, 36, 40], while several studies noted educational resources for patients (e.g. brochures, videos, and staff scripts) were necessary to facilitate completion [9, 34, 36, 41].

Ensuring leadership buy-in [4, 7, 8, 27, 28, 36,37,38, 41], and alignment to the organisation’s strategic goals [7, 8, 36], were both noted to assist in enabling effective ePROM/ePREM implementation. In contrast, stakeholders raised concerns about organisational policies being either lacking entirely or conflicting with ePROM implementation goals [7, 8, 36]. For example, a lack of congruency between organisational policies and implementation goals was identified in Schepers, et al. [7], with a lack of “formal ratification by management” to align policy and implementation work plans. When effectively implemented, ePROMs/ePREMs were noted to allow for comparative analysis within or between systems and organisations [4, 25, 32, 37], and the ePROM/ePREM data could be used to justify continued or expanded funding towards health services [25].

CFIR domain: inner setting

The compatibility of ePROM/ePREMs with the existing workflow processes is mixed. Many studies have noted the benefits of organisational workflow and time management [24, 26,27,28, 34, 35, 39, 42], particularly when ePROMs/ePROMs integrated seamlessly into existing workflow processes [4, 7, 9, 27,28,29,30,31, 33, 35,36,37, 41, 43], and when clinicians are able to access data in real time [4, 28, 30, 32, 33, 36, 39, 43]. A small number of studies have also noted the benefits of providing tablet computers to collect ePROMs at first contact with the patient (usually in the waiting room) [9, 36, 40, 41]. Factors that challenge compatibility primarily relate to the additional workload burden placed on staff [7, 9, 30, 31, 35, 36, 39, 41]. More specifically, concerns were raised around the logistics of ePROMs collection with respect to time and equipment limitations [35, 38, 41, 43], too many ePROMs/ePREMs to collect [25, 30, 31, 33, 35, 38, 39, 41], and the coordination of handing out, retrieving, and cleaning tablet devices [9, 40]. Additional barriers highlighted were with respect to staff turnover and the requirement of training new staff in use of ePROMs [7, 8], where suggested counter strategies included direct access to technical support staff [28, 36], regular staff training and education to build capacity and confidence [8, 9, 24, 27,28,29,30, 33, 35,36,37, 39, 43], and staff or volunteers to facilitate ePROMs collection [7, 28,29,30,31, 33, 34, 36, 37, 39].

Tension to change was identified by clinician stakeholders as an issue leading to anxiety and resistance to change as barriers to ePROM/ePREM implementation [25, 29, 36, 43]. A lack of staff incentives was considered to aid stakeholder resistance [7, 30], however, ePROMs which were easily integrated into existing electronic health records were thought to increase stakeholder readiness and potential for implementation success [7, 8, 32, 33, 35, 36, 38].

Stakeholders viewed leadership buy-in as integral to ensuring a favourable implementation of ePROMs/ePREMs in the clinical setting [4, 7, 8, 27, 28, 36,37,38, 41]. Further facilitators also included the ability to easily share ePROMs/ePREMs results with other clinicians [25], along with the presence of peers who are more familiar with ePROMs, enabling peer-to-peer learning [28, 29, 31, 34, 36, 39]. This was considered to encourage a cultural shift for clinicians to place similar value on ePROMs as other clinical data [41]. Secondary uses of ePROMs/ePREMs data, such as aiding clinical performance metrics [4, 25, 36], and supporting clinician self-reflection and peer supervision [4, 29], have also facilitated implementation.

CFIR domain: characteristics of individuals

There have been mixed findings regarding individuals’ attitudes towards and values placed on ePROMs and ePREMs. Many studies have identified the knowledge and belief systems of individual clinicians as implementation barriers, including lack of knowledge of ePROM content [7, 8, 30, 39, 43], uncertainty in how ePROMs/ePREMs can inform clinical decisions [7, 9, 25, 29, 30, 32, 33, 35, 41, 43], and feelings of being overwhelmed by the excessive volume of reported ePREM data [4]. Additionally, clinicians beliefs that ePROMs/ePREMs are not clinically valid or lack accuracy [24, 25, 28, 30, 38], and are not suitable or relevant or valuable [7, 8, 25, 30, 34, 35, 38, 39, 43], duplicate the clinical interview [8, 25, 30, 38, 43] or fall outside the clinical scope of practice [25, 43], were further implementation barriers. Nonetheless, stakeholders noted utilising ePROM results in clinic visits could validate clinical perceptions about patient outcomes [29], increase the sense of objectivity that clinicians have when communicating with patients [24, 33], and improve prioritisation and targeting of patient–clinician communication [44]. The buy-in of clinical staff was also noted by several studies to influence implementation outcomes [9, 31, 33, 37, 38].

CFIR domain: process

Several studies have noted the benefit of having project managers or coordinators who were skilled in both knowledge translation and facilitating practice changes [8, 35, 36], coupled with the importance of pre-implementation planning and testing of ePROM usability [28, 33, 36]. In turn, the engagement of multiple stakeholders throughout the planning and implementation phases was considered a key to success [8, 27,28,29, 32, 33, 36,37,38]. Furthermore, the presence of local staff champions to support/motivate peers and advocate for ePROMs/ePREMs usage was an important facilitator [4, 8, 28, 30, 31, 33, 34, 36]. The use of standardised workflow processes, in conjunction with regular audit and feedback, enabled the identification and refinement of such processes in early implementation phases, supporting the scalability of ePROMs to additional health facilities [7, 8, 28, 36].

Discussion

This systematic review synthesised literature pertaining to the facilitators and barriers that influence the effective implementation of electronic patient-reported outcomes and experience measures within existing clinical health settings. As identified in previous systematic reviews investigating ePROM implementation [3, 14], the facilitators and barriers highlighted in this review need to be contextualised within the local setting where the ePROMs/ePREMs are to be implemented. However, mapping the results to a theoretical framework such as CFIR, a common language and structure is established for organising the findings, which enables a more comprehensive and structured understanding of the factors that influence the success of an implementation.

This review highlighted that the point-of-care utilisation of ePROMs/ePREMs could strengthen the patient’s voice through improved communication and a focus on shared decision-making in the patient’s healthcare journey. Furthermore, this review builds on two previous systematic reviews, one of which examined the implementation of PROMs (not ePROMs specifically) in a variety of health settings and mapped facilitators and barriers to the CFIR framework [14], whereas, the other reviewed the benefits and disadvantages of ePROMs, without a specific focus on implementation [3]. In contrast, the current review focused specifically on understanding the factors that influence the implementation of ePROMs/ePREMs in clinical healthcare settings.

The data on ePREMs, though limited to just two studies [4, 25], suggests a great deal of overlap between the barriers and facilitators to implementing ePROMs and ePREMs. Specifically, 27 of 30 implementation factors identified within the two included studies investigating ePREMs were also found to be identified in the other studies investigating ePROMs. This degree of overlap exists despite ePREMs having quite different aims to ePROMs, measuring experiences rather than health outcomes. Nonetheless, the paucity of data highlights that ePREMs are a relatively under-utilised measure in the health landscape [4] and underscores the need for further research in this area.

When comparing the frequently mentioned code categories across the five CFIR domains found in this review with previous reviews a moderate number of categories were previously reported [3, 14, 44]. Previously identified facilitators included ePROMs/ePREMs amplifying patients’ voices, facilitating patient-centred care, and shared decision-making [14]; ePROMs allowing patients to better communicate and prioritise in clinic visits [14]; engagement/involvement of stakeholders; pre-implementation testing, especially usability [14]; and project managers/coordinators skilled in knowledge translation and facilitating practice change [14]. Barriers that were already identified in previous reviews included ePROMs as a time burden [14], lack of reliable and robust software and hardware [14], difficulty in completing ePROMs/ePREMs for patients with low language and computer literacy [3, 14, 44], burden on staff facilitating ePROM collection [14], time consumption and too many ePROMs/ePREMs [14, 44], not sure how ePROMs/ePREMs can inform clinical decisions [14], believing ePROMs/ePREMs not suitable/relevant/valuable [44], belief that ePROMs/ePREMs are not clinically valid or lack accuracy [14, 44], and clinicians’ lack of knowledge and content of ePROMs/ePREMs [14, 44]. Several bidirectional categories between previous reviews and the current review included: patients not aware of the purpose of ePROMs; need to have rationale explained to them [14, 44]; regular training and education to build staff capacity and confidence with the ePROM system [14, 44]; integrating ePROMs/ePREMs into existing workflow routine or reconfiguring workflow to ensure integration of ePROMs [14, 44]; improved prioritisation and targeting of patient–clinician communication [3, 14]; and the presence of local staff champions to support/motivate peers and advocate for PROM usage [14, 44].

When looking further afield within the existing literature, a major barrier remains in the integration and workflow of ePROMs/ePREMs into existing (or planned) electronic medical records which can significantly impact the effectiveness of implementation [45, 46]. Additionally, Gensheimer et al. [45], highlighted the importance of integration driven by health system leadership and supported by IT specialists. Briggs et al. identified a barrier arising from a lack of infrastructure, further reinforced by the lack of space to administer ePROMs/ePREMs, user-friendly electronic medical records for ePROM/ePREM integration, and equipment and resources [44].

This study provides unique insights into the implementation of ePROM/ePREMs, beyond those found in previous reviews. Approximately one-third of the most commonly occurring (top five) code categories in each domain of CFIR in the current study were not found in prior reviews, likely as a result of this review being inclusive of more recent literature and specifically focused on ePROMs and ePREMs. The new code categories were primarily mapped to the first two CFIR domains (intervention characteristics and outer setting) because ePROMs/ePREMs differ primarily in a key intervention characteristic—being electronic—and this has implications for how patients (outer setting) interact with them. With the growing application of technology in healthcare and, more directly, the adoption of electronic medical health records, these factors are now more evident in the reviewed literature.

Collectively, these findings highlight and contextualise facilitators and barriers to guide the implementation of ePROMs/ePREMs from the perspective of clinical practice. Recommendations for the implementation of ePROMs/ePREMs were provided as an implementation checklist (Additional File 4: Table S2). The use of such a checklist may ensure implementation efforts are targeted at ensuring the acceptance and sustainability of practice. This, in turn, supports the continued growth of embedding ePROM/ePREM within mainstream healthcare service provision through contextualised implementation across many Australian health sectors [47,48,49]. The use of the constructed implementation checklist from this review’s findings is thought to allow for greater utilisation of ePROMs/ePREMs in local health settings to better enhance patient-centred care and shared decision-making.

This systematic review has both strengths and limitations. First, most studies (n = 15) were conducted in three countries: the USA, Canada, and the Netherlands, whereas half (n = 12) of the studies implemented ePROMs/ePROMs in the clinical areas of either oncology [8, 33, 34, 36,37,38,39, 42] or orthopaedics [9, 30, 41]. As health care provision and funding vary vastly across the world, generalisability of findings to healthcare settings within other countries and other clinical cohorts may be limited. Further research across various countries, funding environments, and clinical populations are required.

Second, only two of the included studies investigated ePREM implementation [4, 25], the findings of this review with respect to implementation of ePREMs must be interpreted with caution because these identified factors are based on far fewer studies than those identified as influencing implementation of ePROMs. It should also be noted that although MMAT quality scores are presented as overall scores, this has been discouraged by some authors [17]. Nonetheless, it is anticipated the inclusion of study characteristics will enable transparency in the interpretation of the ratings. A further strength of this review is that the two reviewers independently synthesised the data, including independent consensus coding. Additionally, the CFIR determinant framework was used as a conceptual map in which code categories could be organised to produce actionable findings, thus enabling a better understanding of where efforts need to be directed when attempting to implement ePROMs/ePREMs. Furthermore, the pragmatic selection and design of strategies to implement the developed checklist by health services and organisations may be assisted using other frameworks (e.g., Expert Recommendations for Implementing Change Matching Tool) [50].

Conclusion

This review provides a contemporary overview of the facilitators and barriers that influence the successful implementation of ePROMs and ePREMs in various health care settings. It highlights several factors that should be considered in future organisational processes when implementing ePROMs/ePREMs. We anticipate the findings of this review will be informative for clinical practitioners, public health officials, and other developing mechanisms for enhancing patient-reported outcomes. Further investigation of the facilitators and barriers identified in this review should be applied to other clinical areas to examine the generalisability of the findings. The relationship between clinician resistance to change in implementing ePROMs/ePREMs and their impact on the subsequent implementation of patient care and satisfaction also requires further investigation.