Abstract
Background
Learning health systems strive to continuously integrate data and evidence into practice to improve patient outcomes and ensure value-based healthcare. While the LHS concept is gaining traction, the operationalization of LHSs is underexplored.
Objective
To identify and synthesize the existing evidence on the implementation and evaluation of advancing learning health systems across international health care settings.
Methods
A mixed methods systematic review was conducted. Six databases (CINAHL, Embase, Medline, PAIS, Scopus and Nursing at Allied Health Database) were searched up to July 2022 for terms related to learning health systems, implementation, and evaluation measures. Any study design, health care setting and population were considered for inclusion. No limitations were placed on language or date of publication. Two reviewers independently screened the titles, abstracts, and full texts of identified articles. Data were extracted and synthesized using a convergent integrated approach. Studies were critically appraised using relevant JBI critical appraisal checklists.
Results
Thirty-five studies were included in the review. Most studies were conducted in the United States (n = 21) and published between 2019 and 2022 (n = 24). Digital data capture was the most common LHS characteristic reported across studies, while patient engagement, aligned governance and a culture of rapid learning and improvement were reported least often. We identified 33 unique strategies for implementing LHSs including: change record systems, conduct local consensus discussions and audit & provide feedback. A triangulation of quantitative and qualitative data revealed three integrated findings related to the implementation of LHSs: (1) The digital infrastructure of LHSs optimizes health service delivery; (2) LHSs have a positive impact on patient care and health outcomes; and (3) LHSs can influence health care providers and the health system.
Conclusion
This paper provides a comprehensive overview of the implementation of LHSs in various healthcare settings. While this review identified key implementation strategies, potential outcome measures, and components of functioning LHSs, further research is needed to better understand the impact of LHSs on patient, provider and population outcomes, and health system costs. Health systems researchers should continue to apply the LHS concept in practice, with a stronger focus on evaluation.
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Introduction
Learning health systems (LHSs) were first defined by the Institute of Medicine in 2007, as a system where “science, informatics, incentives, and culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the delivery process and new knowledge captured as an integral by-product of the delivery experience” [1]. This approach to health system restructuring provides a promising opportunity to enhance value-based healthcare (VBHC). Value-based healthcare (VBHC) places patients at the forefront of health care services, while emphasizing quality of care over number of healthcare interactions [2, 3]. This approach also aims to reduce costs without sacrificing value [2, 4]. LHSs also place patients at the centre of the health system, with continuous learning from patient experience and outcome data cycling back into the system to improve care [5]. The LHS concept therefore aligns with the goals of VBHC [6]. There has been a global shift towards VBHC, and this concept is now recognized as a top health system priority [7, 8]. Despite the opportunity for LHSs to achieve VBHC across health systems, there are gaps in our understanding of how to operationalize LHSs.
Since its inception in 2007, the idea of LHSs has evolved to include several descriptions and features. Lavis et al. (2018) proposed seven characteristics reflective of a LHS: (1) engaged patients; (2) digital capture, linkage and timely sharing of relevant data; (3) timely production of research evidence; (4) appropriate decision supports; (5) aligned governance, financial and delivery arrangements; (6) culture of rapid learning and improvement; and (7) competencies for rapid learning and improvement [9]. In 2019, Menear and colleagues developed a LHS framework comprised of four key elements: (1) core values; (2) pillars and accelerators; (3) processes; and (4) outcomes [6]. The framework presents a structure in which health systems can work towards delivering more VBHC [6]. Another review by Zurynski and colleagues (2020) included over 200 LHSs papers and reported on the LHS terminology, frameworks, barriers and enablers of LHSs across the literature [10]. Studies in this scoping review used varying terms to describe LHSs, including learning health networks, rapid learning systems and learning healthcare systems [10]. Clearly, LHSs are gaining traction as a valuable model for healthcare organizations, and despite the varied terminology, the central focus on rapidly incorporating evidence into practice to enhance VBHC remains consistent.
While there is ample literature describing LHS characteristics, there is little information on how to put this model into practice. Effective implementation of LHSs has the potential to improve patient outcomes, reduce costs and enhance quality care [6]. So, without a proper understanding of LHS implementation, research and health system resources are lost. A narrative review of LHS frameworks by Allen et al., identified a roadmap to assist organizations in creating LHSs [11]. The authors presented a logic model with key inputs, outputs and outcomes, based on the core features of 17 LHS frameworks and models [11]. This study is a valuable resource to help health systems move towards a LHS model, however, this roadmap has not yet been applied to LHS-focused studies. Further, to date, no reviews have explored the types and outcomes of implementation strategies used by existing LHSs. As such, while the idea of LHSs is promising, it is still unclear how LHSs are operationalized across different health care organizations and countries. The scoping review by Zurynski et al. identified several functioning LHSs, but they did not describe how these LHSs were implemented or the outcomes of the implementation process [10]. Evidently, there is a need to unpack and synthesize the implementation process of LHSs across health care settings.
Implementation science is a field of research focused on methods and strategies to facilitate the uptake of evidence-based intervention and policies. Implementation strategies are techniques used to support the effective uptake of an intervention [12]. The Expert Recommendations for Implementing Change (ERIC) taxonomy includes a comprehensive list of 73 implementation strategies that were developed based on a review of the evidence and expert consultation [12]. Implementation scientists often apply these strategies in their research to ensure an intervention is delivered in a systematic, evidence-based way. Further, measures related to the implementation process can provide helpful information about whether an intervention led to meaningful change [13]. Tierney et al. established a list of 10 implementation measures specifically for evidence synthesis studies that provide additional insight as to the value of an intervention or research project [13]. While both the ERIC taxonomy and Tierney’s implementation measures have been applied to previous implementation research, they have not yet been applied to LHS-focused evidence synthesis work. With a clear gap in the evidence related to LHS implementation, there is an opportunity to understand how LHSs have been designed. Therefore, the aim of this study is to systematically synthesize the evidence on the implementation of LHSs across different health care organizations and countries. This aim will be achieved through the following objectives:
-
Describe the LHS characteristics used across studies and health care organizations
-
Identify the number and types of implementation strategies used to transition to a LHS
-
Describe the LHS outcome measures applied across studies
Methods
Study design
A mixed methods systematic review (MMSR) was conducted following the Joanna Briggs Institute (JBI) methodological guidelines for MMSR [14]. A MMSR allows for the comprehensive overview of a broad research question or phenomenon of interest and may include evidence from qualitative, quantitative, and mixed methods study designs.
This review was registered in PROSPERO (CRD42022293348) and the protocol was previously published [15]. The Preferred Reporting in Systematic Reviews and Meta-Analyses (PRISMA) checklist was used to report the findings of this study [16].
Inclusion criteria
Following JBI guidelines for MMSR [14], the PICo (Population; Phenomenon of Interest; Context) framework guided the question development and identification of inclusion criteria. This review aimed to answer the following research question: How do healthcare organizations implement and evaluate the transformation of learning health systems?
Population
Studies were considered for inclusion if they described a LHS, including rapid learning systems, rapid learning healthcare, learning healthcare systems, learning health systems or other similar LHS terms. Due to the inconsistency in reporting on LHSs, studies which described components of a LHS without using LHS terminology were excluded.
Phenomenon of Interest
This review included studies reporting on implementation strategies and/or outcome measures associated with the adaptation of LHSs. Implementation strategies include any procedure, approach, or method to implement, assess or evaluate the uptake of LHSs. The ERIC taxonomy of 73 strategies was used to identify implementation strategies reported across studies [12]. Any reported outcome measures were identified using Tierney et al.’s list of 10 implementation measures [13] to provide further insight into the value of the intervention and implementation process. Studies were further tagged as either provider, patient, population or healthcare cost-related outcomes, to reflect the quadruple aim of enhancing health systems [17, 18].
Context
Studies conducted in any health care setting were included. Health care settings may include hospitals, academic medical centres, primary care clinics, community health centres, practice-based networks or individual departments or clinics that provide health care to patients. Any country and size of healthcare organization were considered for inclusion. Non-healthcare settings, such as academic institutions, government, or non-government organizations where care is not directly provided to patients, were excluded.
Types of studies
Quantitative, qualitative, and mixed methods studies were considered for inclusion in this review. Additionally, descriptive papers reporting on the implementation of LHSs were included if they were peer reviewed. Grey literature sources such as policy reports, case studies or conference proceedings were included if they described the implementation strategies and/or outcome measures of LHSs. Protocol papers were excluded but forward citation searching was conducted to find any published studies stemming from the protocol. Similarly, reviews were excluded but the reference lists of identified reviews were manually searched for relevant papers.
Search strategy
Six databases were searched up to July 2022, for key terms related to LHSs, implementation and health care, using a comprehensive search strategy developed by a research librarian trained in knowledge synthesis. The search strategy was peer reviewed (PRESSed) by an independent research librarian to validate the approach. The databases included CINAHL (EBSCOhost), Medline (Ovid), Embase (Elsevier), Nursing and Allied Health Database (ProQuest), PAIS (ProQuest) and Scopus (Elsevier). Boolean operators and MESH terms were used accordingly for each database. An example search strategy for CINAHL can be found in Additional file 1: Table S1. No restrictions were placed on date of publication or language. A grey literature search was conducted to identify additional relevant articles. This involved searching ProQuest’s Dissertations and Theses Global, a targeted search of the websites of three pre-identified relevant organizations, and a systematic Google search to identify relevant sources. The grey literature search strategy can be found in Additional file 1: Table S2 while a comprehensive description of the search strategy can be found in the published protocol paper [15]. Additional articles were retrieved through backward and forward citation searching of reference lists of included articles.
Study selection
Identified articles were uploaded to the data management software, Covidence (Veritas Health Innovation, Melbourne, Australia), and duplicates were removed electronically. Two reviewers (MS and CJ) independently screened the titles, abstracts and full texts of identified articles based on the predetermined inclusion criteria. Any discrepancies in screening decisions were resolved through discussion by the reviewers, with an independent, third reviewer (CC), helping to reach consensus as needed. Studies were deemed eligible for inclusion if they reported on the implementation of LHSs, as outlined in the predetermined inclusion/exclusion criteria. One reviewer screened the first five pages of the returned grey literature search results. Relevant articles were uploaded to Covidence and followed the same screening approach as the database search results. The screening results, along with reasons for exclusion, were reported in the 2020 PRISMA flow diagram [16].
Data extraction
Following the screening stage, data were extracted from each included study using a pre-determined, data extraction form. The data extraction process was independently pilot tested by three reviewers (CJ, MS and DS) on a sample of included articles (n = 5). Data extractors met to discuss discrepancies in the data extraction process and changes were made to the extraction sheet as needed. Data were then extracted from the included articles by one reviewer (CJ) and verified by a second reviewer (MS) to ensure consistency and reliability of results. Weekly team meetings were held during the data extraction phase to discuss any arising concerns with the included articles.
Extracted data included study characteristics such as country, year of publication, study design, objective, population and description of LHS. Any reported details about implementation were also extracted. Qualitative findings were extracted as themes and sub-themes and included concepts related to implementing LHSs, such as stakeholder experiences in how healthcare organizations shifted to a LHS model. Evaluation measures and outcome data were also extracted from qualitative studies when available. Quantitative data related to the implementation and/or evaluation of LHSs were extracted. Quantitative findings included patient-related outcomes, cost effectiveness, provider outcomes, pre-post data, changes in population health and impact on the health system.
Data synthesis and integration
The LHS details from each study were synthesized using Lavis et al. seven LHS characteristics [9]. For example, studies reporting on patient engagement were tagged as such, and an overview of the most and least common LHS characteristics were reported narratively. Similarly, the implementation strategies described by authors were synthesized and categorized using the ERIC taxonomy of implementation strategies [12]. A descriptive synthesis of the most common strategies was reported. Outcome and evaluation details were also synthesized by mapping the reported outcome measures to Allen et al.’s list of 10 outcomes for LHSs [11], while Tierney’s list of implementation outcome measures was used to further categorize how studies reported implementation outcome measures [13]. These details provide a comprehensive picture of how LHSs have been implemented across various health care settings and whether the implementation of LHSs led to changes in health care costs, patient, provider, or population level outcomes.
A convergent integrated approach was used to synthesize the data in this review. This approach involves extracting quantitative and qualitative data separately, followed by an integrated synthesis of all sources of data [14]. First, the findings from quantitative studies were transformed into ‘qualitized’ data. This involved creating a narrative description for each quantitative study's key findings, by extracting the key findings from each study and then reviewing and refining them by two independent reviewers. A similar approach was used with qualitative study findings from included qualitative and mixed methods papers. The final, agreed upon narrative description of key findings from all studies were then integrated using thematic analysis by categorizing and pooling similar findings together. This involved two reviewers independently coding all findings, and then determining a list of common themes. The themes were then refined and finalized through further discussion among three authors, until consensus was reached. A final list of integrated findings was then reported in tables and text.
Critical appraisal
Included studies were critically appraised by two reviewers (MS and CJ) using the relevant JBI critical appraisal tool, according to study design. The purpose of the tools is to appraise different types of study designs, to provide an objective summary of the design quality. JBI is a reputable organization that specializes in access, appraisal and application of the best available evidence for evidence-based decision making in health and service delivery. The appraisal tool questions can be found in Additional file 1: Table S3. Qualitative studies were appraised using the qualitative checklist and scored out of 10. Cross-sectional studies were scored out of eight, cohort studies were scored out of 11 and quasi-experimental studies were scored out of nine. The reviewers independently appraised each study and then met to discuss final scores. In instances of differently scored items, the reviewers met to discuss their scores until a consensus was reached. A third reviewer was consulted in cases where a decision could not be reached. Final scores were presented as a percentage alongside extracted data in tables, with a detailed overview of study scores presented in a separate table. Grey literature sources and descriptive case studies were not appraised due to a lack of a relevant appraisal tool, and therefore did not receive a critical appraisal score. This was documented in the results tables. In line with mixed methods systematic review methodology [14], no confidence of findings summary table was developed. Due to the heterogeneous nature of mixed methods reviews, it is not recommended to complete this step.
Results
A total of 5171 studies were identified in the database search, of which 3147 were removed as duplicates. Of the remaining 2024 studies, 27 were deemed relevant and one additional study was found through hand searching reference lists [19]. The grey literature search returned over 700 000 resources, and after reviewing the first five pages of each search, 40 were screened in full text and seven were included in this review. Therefore, a total of 35 resources, describing 31 unique LHSs, were included in this review.
The main reasons for exclusion included wrong article type (n = 78), not related to implementation (n = 69), not about LHSs (n = 49), wrong study design (n = 39), duplicate study (n = 9) or not a healthcare setting (n = 1). A complete list of the search process can be found in the PRISMA flow diagram (Fig. 1).
PRISMA flow diagram of included studies. *Consider, if feasible to do so, reporting the number of records identified from each database or register searched (rather than the total number across all databases/registers). **If automation tools were used, indicate how many records were excluded by a human and how many were excluded by automation tools.From: Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi: 10.1136/bmj.n71. For more information, visit: http://www.prisma-statement.org/
Study characteristics
Of the included studies, the majority were conducted in the United States (US) (n = 21) followed by Canada (n = 4), the United Kingdom (UK) (n = 3), Australia (n = 1), Sweden (n = 1), Netherlands (n = 1), and Europe (n = 1). Four studies reported on LHSs implemented across international borders. The date of publication ranged from 2014 to 2022, with most studies (24/35) published between 2019 and 2022. Of the 35 included studies, five were of qualitative design [20,21,22,23,24], four were cross-sectional studies [25,26,27,28], two were cohort studies [29, 30], one was quasi-experimental [19] and 23 were descriptive case studies [31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52]. All five of the qualitative studies involved semi-structured interviews and reported on stakeholder views related to implementing LHSs. Together, the qualitative studies described 20 themes related to implementing LHSs. The remaining quantitative and descriptive case studies varied in their designs and outcomes of interest. The case studies described the implementation of LHSs, without reporting on a specific methodological approach to data collection and analysis. The type of health setting varied across studies, with the majority of LHSs implemented in hospitals at a multi-institutional level, often for a specific health condition. The study population varied across studies, with 11 reporting on a pediatric population [26, 28, 29, 35,36,37, 39, 41, 44, 46, 52], 16 reporting on health system leaders or employees [20,21,22,23,24,25, 27, 31,32,33,34, 43, 45, 49, 50, 53] and eight related to adult populations with various clinical presentations [19, 30, 38, 40, 42, 47, 48, 51]. Study characteristics can be found in Table 1.
Learning health system characteristics
Synthesis of LHS characteristics revealed that of the 31 unique LHSs (reported across 35 studies) eight included all seven of Lavis et al. characteristics [35, 36, 41, 42, 44, 45, 47, 53]. Two studies [25, 52] only reported two LHS characteristics, and these included the ‘digital data capture’, ‘timely production of evidence’ and ‘appropriate decision supports’ characteristics. Digital data capture was reported most often, in 31 studies. Examples of this characteristic included when a LHS incorporated an electronic health record (EHR) as part of the system or the use of dashboards and databases to enable data sharing. Patient engagement, aligned governance, and the culture of rapid learning and improvement characteristics were reported least often, in 21 studies each. Table 2 outlines the number and types of LHS characteristics reported across studies. A more detailed description of the LHS constructs identified in each study can be found in Additional file 1: Table S4.
Implementation strategies
All study authors reported on the implementation of a LHS, including the specific implementation strategies used, and/or implementation outcomes. Of ERIC’s taxonomy of 73 strategies, 33 different implementation strategies were used across studies. The most common implementation strategies were change record systems (n = 20) [22, 26, 28,29,30,31,32,33,34,35,36, 38, 40, 45, 46, 48, 50,51,52,53], conduct local consensus discussions (n = 7) [19, 28, 29, 32, 37, 39, 49], audit and provide feedback (n = 6) [19, 22, 25, 43, 50, 53], build a coalition (n = 5) [20, 26, 41, 44, 46], and develop and organize quality monitoring systems (n = 5) [21, 27, 35, 44, 46]. Fourteen strategies were reported only once across studies. Table 3 provides an overview of the reported implementation strategies with an example from select papers.
Study outcomes
Systematic adoption of evidence-based practices (EBP) was the LHS outcome reported most often (n = 17) according to Allen et al.’s list, followed by knowledge to action latency (n = 12) and population health (n = 12). Using Tierney’s list of implementation measures, the majority of studies commented on intervention complexity (n = 16) and adoption (n = 13). Only two studies spoke about implementation cost [28, 29] and no studies discussed fidelity as part of their LHS implementation approach. Based on the quadruple aim (patient, provider, population and health care cost), 23 studies reported on outcomes related to patients [19,20,21, 24, 26, 28,29,30,31,32,33,34,35,36, 38, 39, 44, 46,47,48, 50, 52, 53], eight studies addressed provider outcomes [20, 22,23,24,25, 27, 39, 43], 15 studies were related to population-level outcomes [26, 27, 36,37,38,39,40,41,42, 45, 46, 48, 49, 51, 53] and five studies reported on healthcare costs [28, 29, 31, 34, 35]. Table 4 provides an overview of study outcomes.
Integrated findings
The key findings reported across studies were heterogeneous in nature and therefore a meta-analysis was not possible. Rather, the main findings from all studies were integrated and thematically organized. Quantitative outcome data was qualitized and pooled along with the qualitative and descriptive study outcomes to reveal three main integrated findings and six sub-findings. These integrated findings are described in text below and in Table 5.
Integrated finding 1: the digital infrastructure of LHSs optimizes health service delivery
Multiple studies reported on how the implementation of a LHS impacted health service delivery, such as by allowing for the rapid inclusion of evidence into practice or by providing an infrastructure to support digital data capture. Three categories were grouped under this main finding. The first describes how LHSs can allow for better integration of data and evidence into clinical practice. Fifteen studies aligned with sub-finding 1a and reported on how the implementation of a LHS can allow for better integration of data and evidence into practice, such as having a platform that highlights chronic pain in patients to inform care [26] or a database with information about patients with multiple sclerosis that clinicians use to inform decisions [30]. Sub-finding 1b includes how LHSs promote the implementation of digital data capture. Sixteen studies described how a LHS can provide an opportunity for digital data capture [19, 21,22,23,24, 31, 32, 39,40,41, 44,45,46,47, 50, 51]. In some cases, the digital infrastructure was not available or valued prior to the implementation of a LHS, but with the health system transformation, systems were able to see how infrastructure could support the collection of digital data. This was often viewed as key to the LHS implementation process. Sub-finding 3c describes that access and availability of data through a digital platform supports rapid changes to practice and policy. Seven studies aligned with this finding and described how the LHS allowed for faster changes to practice and policy [25,26,27,28,29, 33, 38]. In some studies, this was a result of having electronic data more readily available to practitioners, allowing for rapid decisions and changes to patient care.
Integrated finding 2: learning health systems have a positive impact on patient care and health outcomes
The second integrated finding is represented by 17 studies [19, 22, 28, 29, 32, 33, 35,36,37,38,39, 42, 43, 45, 49, 51, 52] which reported positive patient outcomes as a result of implementing a LHS. These studies included varied patient outcomes such as improved diagnosis, screening or referral rates [38], reduced readmission rates and decreased length of stay [29], and improved prescribing practices [19, 39, 51, 53].
Integrated finding 3: learning health systems can influence health care providers and the health system
The third main finding is related to the impact of LHSs on the health system, including the physical environment and people within the system. This finding included three sub-findings. The first sub-finding, 3a, states how implementation of a LHS may help foster a culture of learning and improvement for sustained success. Eight studies related to this finding and described how the LHS changed the culture within their healthcare organization [21, 23, 31, 33, 34, 39, 43, 45]. This included having a stronger culture of continuous learning and improvement, with several studies describing this feature as being crucial to LHS sustainability overtime. Sub-finidng 3b describes how health system leaders identify challenges in implementing LHSs, despite recognizing their value. Four studies aligned with finding 3b and reported the challenges of implementing a LHS [20, 24, 47, 50], such as facing financial barriers to implement and sustain the digital infrastructure [24] or in getting buy-in from key stakeholders [20]. Sub-finding 3c includes how LHSs may result in cost savings for the health system. Five studies talked about the potential cost savings of LHSs, following an economic analysis of their respective LHSs [28, 29, 31, 34, 35].
Methodological quality
Of the 35 included studies, 12 were appraised for methodological quality [19,20,21,22,23,24,25,26,27,28,29,30]. The remaining 23 papers were grey literature or descriptive case studies and were therefore not eligible for appraisal. Studies were appraised using the relevant JBI appraisal tool and a score was assigned based on percentage of criterion met in each study. Quality scores ranged from 25 to 91%, with five studies receiving a score of 75% or higher [20, 21, 26,27,28]. Table 6 provides an overview of study scores for the 12 appraised studies.
Discussion
Learning health systems offer a promising approach to advance VBHC; however, it is unclear how LHSs are operationalized across different health care organizations and countries. Our mixed methods review addresses this gap by highlighting and synthesizing the types of implementation strategies most used when transitioning to a LHS and provides some outcome data from functioning LHSs. Researchers and health system leaders may use these findings to support their own LHS implementation and evaluation efforts.
LHS characteristics and implementation strategies
Studies highlighted the importance of digital infrastructure to capture data and integrate it back into the system to inform decision-making. This is a central component of LHSs described across the literature [6, 9, 54], and it is not surprising that this was a key finding in our review. Digital data capture was included as part of the LHS description in 31 studies, while digital infrastructure to support health services delivery was revealed as a key integrated finding across studies. Further, changing record systems was identified as the most common implementation strategy, highlighting that most health systems adapted their digital infrastructure in some way to implement their LHS. To support other LHS features, such as incorporating patient and provider experiences, and having timely production of evidence, it makes sense that establishing the appropriate digital infrastructure is a preliminary step. This aligns with previous LHS research that found a lack of infrastructure, digital registries and electronic systems for capturing patient data were common barriers to developing a LHS [10]. Clearly, establishing infrastructure for digital data capture is central to LHS implementation and therefore, researchers and health systems leaders may want to prioritize this aspect of implementation for advancing LHS transformation.
Only 21 studies included patient engagement as part of their LHS description, while one study included patient, consumer, and family feedback as an implementation strategy [36]. Another two studies reported involving patients or consumers to enhance the uptake and adherence of the LHS intervention [26, 42]. Evidence suggests that patient-centeredness in healthcare leads to improved patient outcomes and quality of care [55]. However, patient engagement has been lacking in LHS literature, with patient-clinician partnerships cited least often in a synthesis of LHS papers [10], and some LHS frameworks excluding this dimension altogether [54, 56]. While it is crucial not only to engage patients in LHS development, but to incorporate patient experience data back into the system, there are challenges in achieving this. Patients want LHSs to be transparent about the use of their data [57] and to have open communication and shared decision-making with their clinical provider within the LHS [58]. Patient-centred care should be central to any healthcare system, including LHSs. However, more research is needed to understand how to best engage patients in the development, implementation, and evaluation of LHSs to ensure health system transformation remains patient-centered.
Along with patient engagement, aligned governance and a culture of rapid learning and improvement were reported the least often (in 21 papers each). Only eight studies [35, 36, 41, 42, 44, 45, 47, 53] touched on all seven LHS characteristics in their study description. While it is possible that authors were not comprehensive in their LHS reporting, the findings from this review clearly indicate that key components of LHSs are not consistently reported across the literature. Aligned governance is a challenging feature to implement in LHSs [10, 59], largely due to ethical issues in policies and regulations [10, 60]. There are also barriers to establishing a culture of learning and improvement in LHSs due to low buy-in from health system stakeholders [35, 61]. This was a key integrated finding of this review, reported in multiple studies [20, 24, 47, 50]. Regardless, this aspect of the LHS was also recognized as important for ensuring it would be sustained overtime [43, 48, 49, 52]. Clearly, all LHS characteristics are important but there are challenges in incorporating each feature, which means some LHSs are missing key components. An evaluation framework or checklist would be useful to address these challenges in LHS implementation and to ensure researchers are meeting the requirements of a fully functioning LHS.
In this review, we used ERIC’s taxonomy of 73 implementation strategies to code how LHSs were implemented in health service organizations. Of the 73, only 33 distinct implementation strategies were employed, with changing record systems cited as the most common. Much of the literature highlights the need for a culture of rapid learning and improvement to facilitate LHS transformation [21, 49, 62]. Despite this critical component, few studies identified in our review employed implementation strategies that would adequately target culture change. While changing record systems are required to support the data linkage component of a LHS, LHSs will never be fully realized unless strategies aim to facilitate a culture of rapid learning and improvement. Future LHS initiatives should consider building on the strategies identified in this review including conducting local consensus discussions and building a coalition. Future LHS research should test additional implementation strategies that have not yet been evaluated for LHS implementation, including strategies that facilitate the development of stakeholder interrelationships and support clinicians to engage in LHS activities [63].
LHS outcomes
Of the 35 studies in this review, 17 reported positive patient outcomes following implementation of their LHS. This finding is promising as one of the main goals of a LHS is to achieve VBHC, including improving patient outcomes and providing better quality care [2]. However, it is unclear what mechanisms directly led to improved patient outcomes and whether certain LHS features are more strongly associated with positive outcomes. Few studies reported on outcomes related to provider, population and health system costs and most studies did not evaluate the impact of LHS implementation. There is a need for further evaluation research to explore the full impact of LHSs, including how well it addresses the quadruple aim. Finally, the majority of LHSs in this review targeted a particular health condition or patient population. These LHSs were often conducted at an individual department level or as a multi-institutional network with several condition-specific departments working together. Few studies reported on a LHS at the organization level, such as across an entire hospital. This aligns with findings from a recent review that found only four of 76 studies described a LHS as an entire hospital system [64]. With the concept of LHSs still new, it makes sense that researchers may want to first establish a LHS for a particular patient population before expanding more broadly across an institution. However, there are examples of larger scale LHSs, such as the Swiss LHS, being implemented nationally across Switzerland [65]. It is important for researchers to learn from these broader health system examples to continue to scale and spread the efforts and impact of the LHS model.
Implications for research, practice and policy
This comprehensive mixed methods systematic review illustrates important implications for research and health system leaders. First, this review highlighted the value of having a robust infrastructure to support digital data capture when implementing a LHS. Health system leaders and researchers should to prioritize this aspect of LHSs for an effective transition. Second, efforts are needed to support the engagement of patients into the development, implementation, and evaluation of LHS. Third, although many strategies are described in the implementation science literature, few were described in the included LHS studies. Future research should test additional strategies that facilitate partnerships development and engaging clinicians in LHS. Lastly, this review provides evidence that LHSs can lead to improved patient outcomes but there is a need for further evaluation studies on the overall effectiveness of LHSs and their impact on patient, provider, population and cost-related outcomes. With most LHSs being implemented at either an individual clinical unit or a multi-institutional network for a particular medical condition, there is a need for future implementation research to explore large scale health system transformation, such as organizational (e.g., entire hospital), provincial or state-wide health system transformation.
Strengths and limitations
A key strength of this review is the comprehensive mixed methods approach. With the literature on LHSs continuing to emerge, it was important to capture the broad range of studies related to implementation. The inclusion of grey literature allowed for additional case study examples of LHSs to be explored and synthesized. Several aspects of implementation and LHS characteristics were used to categorize studies, which may be useful for health systems researchers and administrators to understand how they can apply similar approaches in their respective healthcare settings. A limitation of the current study includes the potential for biases in the critical appraisal process. While the studies were appraised by one reviewer and verified by a second, some of the questions in the JBI appraisal tool require interpretation of the authors, which allows for potential bias. Additionally, some of the studies included in this review did not have an appraisal tool that fit the methodology, and thus were unable to be appraised. The inconsistent and evolving LHS terminology was a challenge in this review, as some studies may have been excluded if they did not explicitly refer to their intervention as a LHS. This was necessary to avoid irrelevant papers but may have unfairly excluded studies that were truly LHSs but used lesser-known terminology. While no restrictions were placed on language or country, the majority of included studies were conducted in high-income, English-speaking countries. There is a need to explore LHSs in the context of low and middle-income countries.
Conclusion
In this mixed methods systematic review, we described the implementation of LHSs in various healthcare settings, including implementation strategies, outcome measures, and components of functioning LHSs. As the field of LHS science and practice continues to advance, research is needed to better understand the impact of LHSs on patient, provider and population outcomes, and health system costs. Health systems researchers should continue to apply the LHS concept in practice, with a stronger focus on evaluating implementation strategies and outcomes.
Availability of data and materials
All data generated or analyzed during this study are included in this published article and its supplementary information files.
Abbreviations
- AHRQ:
-
Agency of Healthcare Research and Quality
- ATN/AIR-P:
-
Autism Treatment Network & Autism Intervention and Research Network on Physical Health
- CHC:
-
Connected Health Cities
- CHOIR:
-
Collaborative Health Outcomes Information Registry
- CORE:
-
Center for Outcomes Research and Evaluation
- COVID:
-
Coronavirus disease
- EHR:
-
Electronic Health Record
- EQUIPPED:
-
Enhancing Quality of Prescribing Practices for Older Adults in the Emergency Department
- GERD:
-
Gastroesophageal reflux disease
- GHS:
-
Geisinger Health System
- HCA:
-
Hospital Corporation of America
- IBD:
-
Inflammatory bowel disease
- ICN:
-
Improve Care Now
- IDEA4PS:
-
Institute for the Design of Environments Aligned for Patient Safety
- LFEP:
-
Learn from every patient
- LHS:
-
Learning health system
- MS PATHS:
-
Multiple Sclerosis Partners Advancing Technology and Health Solutions
- MSQC:
-
Michigan Surgical Quality Collaborative
- NPCQIC:
-
National Pediatric Cardiology Quality Improvement Collaborative
- PCICCN:
-
Post COVID-19 Interdisciplinary Clinical Care Network
- OPQC:
-
Ohio Perinatal Quality Collaborative
- Peds-CHOIR:
-
Pediatric adaptation of the Collaborative Health Outcomes Information Registry
- QA:
-
Quality Appraisal
- RCLS-CF:
-
Registry-enabled Care and Learning System for Cystic Fibrosis
- SCK:
-
Sickle cell knowledge base
- SHOnet:
-
Shriners Hospitals for Children (SHC) Health Outcomes Network
- SPS:
-
Solutions for Patient Safety
- SNEPT:
-
Starzl Network for Excellence in Pediatric Transplantation
- UK:
-
United Kingdom
- USA:
-
United States of America
- VA-ESP:
-
Veterans Affairs Evidence Synthesis Program
- VBHC:
-
Value-based healthcare
References
Institute of Medicine (US) Roundtable on Evidence-Based Medicine. The learning healthcare system: workshop summary. Olsen L, Aisner D, McGinnis JM, editors. Washington (DC): National Academies Press (US); 2007.
Porter ME. What is value in health care? N Engl J Med. 2010;363(26):2477–81.
Hurst L, Mahtani K, Pluddemann A, Lewis S, Harvey K, Briggs A, et al. Defining Value-based Healthcare in the NHS: CEBM report [Internet]. University of Oxford; 2019 May. https://www.cebm.ox.ac.uk/resources/reports/defining-value-based-healthcare-in-the-nhs
Zelmer, Jennifer. Aligning outcomes and spending: Canadian experiences with value-based healthcare [Internet]. 2018, p. 60. https://www.cfhi-fcass.ca/docs/default-source/itr/tools-and-resources/vbhc/vbhc-executive-brief-e
Agency for Healthcare Research and Quality. About Learning Health Systems [Internet]. 2019. https://www.ahrq.gov/learning-health-systems/about.html
Menear, Matthew B Marc Andre, Demers-Payette, Olivier R Denis. A framework for value-creating learning health systems. Health Res Policy Syst. 2019;17(79).
World Economic Forum. Value in Healthcare Laying the Foundation for Health System Transformation [Internet]. 2017. www.weforum.org
World Health Organization. Delivering quality health services: a global imperative for universal health coverage [Internet]. https://www.who.int/publications-detail-redirect/9789241513906
Lavis, John G Francois Pierre, Reid, Robert B Heather, Wodchis, Walter P H Anne. Creating a Rapid-learning Health System in Ontario. McMaster Health Forum; 2018. p. 62.
Zurynski Y, Smith CL, Vedovi A, Ellis LA, Knaggs G, Meulenbroeks I, et al. Mapping the learning health system: A scoping review of current evidence. Sydney, Australia: Australian Institute of Health Innovation and the NHMRC Partnership Centre for Health System Sustainability; 2020. p. 63.
Allen C, Coleman K, Mettert K, Lewis C, Westbrook E, Lozano P. A roadmap to operationalize and evaluate impact in a learning health system. Learn Health Syst. 2021;5(4): e10258.
Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10(1):21.
Tierney AA, Haverfield MC, McGovern MP, Zulman DM. Advancing evidence synthesis from effectiveness to implementation: integration of implementation measures into evidence reviews. J Gen Intern Med. 2020;35(4):1219–26.
Lizarondo, L S C, Carrier, J G C, Rieger, K S S, Apostolo, J K P, Loveday, H. JBI Manual for Evidence Synthesis. 2020 [cited 2021 Oct 10]. Chapter 8: Mixed methods systematic reviews. https://jbi-global-wiki.refined.site/space/MANUAL/3283910764/Chapter+8%3A+Mixed+methods+systematic+reviews. Accessed 10 Oct 2021.
Somerville M, Cassidy C, Curran J, Rothfus M, Sinclair D, Elliott RA. What implementation strategies and outcome measures are used to transform healthcare organizations into learning health systems? A mixed-methods review protocol. Health Res Policy Syst. 2022;20(1):97.
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;29(372): n71.
Bodenheimer T, Sinsky C. From triple to quadruple aim: care of the patient requires care of the provider. Ann Fam Med. 2014;12(6):573–6.
Canadian Institutes of Health Research, Institute of Health Services and Policy Research (Canada). Strategic plan 2021–2026: accelerate health system transformation through research to achieve the quadruple aim and health equity for all. [Internet]. 2021. https://epe.lac-bac.gc.ca/100/201/301/weekly_acquisitions_list-ef/2021/21-33/publications.gc.ca/collections/collection_2021/irsc-cihr/MR4-47-2021-eng.pdf. Accessed 14 Nov 2021.
Vaughan CP, Hwang U, Vandenberg AE, Leong T, Wu D, Stevens MB, et al. Early prescribing outcomes after exporting the EQUIPPED medication safety improvement programme. BMJ Open Qual. 2021;10(4): e001369.
Dixon-Woods M, Campbell A, Chang T, Martin G, Georgiadis A, Heney V, et al. A qualitative study of design stakeholders’ views of developing and implementing a registry-based learning health system. Implement Sci. 2020;15(1):16.
Enticott J, Braaf S, Johnson A, Jones A, Teede HJ. Leaders’ perspectives on learning health systems: a qualitative study. BMC Health Serv Res. 2020;20(1):1087.
Jeffries M, Keers RN, Phipps DL, Williams R, Brown B, Avery AJ, et al. Developing a learning health system: Insights from a qualitative process evaluation of a pharmacist-led electronic audit and feedback intervention to improve medication safety in primary care. PLoS ONE. 2018;13(10): e0205419.
Nash DM, Brown JB, Thorpe C, Rayner J, Zwarenstein M. The alliance for healthier communities as a learning health system for primary care: a qualitative analysis in Ontario. Canada J Eval Clin Pract. 2022;28(6):1106–12.
Steels S, Ainsworth J, van Staa TP. Implementation of a “real-world” learning health system: results from the evaluation of the Connected Health Cities programme. Learn Health Syst. 2021;5(2): e10224.
Floyd N, Peterson K, Christensen V, Anderson J. “Implementation Is so Difficult”: survey of national learning health system decision-makers identifies need for implementation information in evidence reviews. Med Care. 2019;57(10 Suppl 3):S233–8.
Bhandari RP, Feinstein AB, Huestis SE, Krane EJ, Dunn AL, Cohen LL, et al. Pediatric-collaborative health outcomes information registry (Peds-CHOIR): a learning health system to guide pediatric pain research and treatment. Pain. 2016;157(9):2033–44.
Miller R, Coyne E, Crowgey EL, Eckrich D, Myers JC, Villanueva R, et al. Implementation of a learning healthcare system for sickle cell disease. JAMIA Open. 2020;3(3):349–59.
Noritz G, Boggs A, Lowes LP, Smoyer WE. “Learn From Every Patient”: how a learning health system can improve patient care. Pediatr Qual Saf. 2018;3(5): e100.
Lowes LP, Noritz GH, Newmeyer A, Embi PJ, Yin H, Smoyer WE, et al. “Learn From Every Patient”: implementation and early results of a learning health system. Dev Med Child Neurol. 2017;59(2):183–91.
Mowry EM, Bermel RA, Williams JR, Benzinger TLS, de Moor C, Fisher E, et al. Harnessing real-world data to inform decision-making: multiple sclerosis partners advancing technology and health solutions (MS PATHS). Front Neurol. 2020;11:632.
Agency for Healthcare Research and Quality,. Baylor Scott & White Health: Building a Foundation for Continuous Improvement [Internet]. 2019 p. 8. https://www.ahrq.gov/sites/default/files/wysiwyg/lhs/lhs_case_studies_bsw.pdf
Agency for Healthcare Research and Quality. Denver Health: How a Safety Net System Maximizes Its Value [Internet]. 2019 p. 7. https://www.ahrq.gov/sites/default/files/wysiwyg/lhs/lhs_case_studies_denver_health.pdf
Agency for Healthcare Research and Quality,. HCA: How a Large Healthcare System Is Looking Beyond the Electronic Health Record [Internet]. 2019 p. 6. https://www.ahrq.gov/sites/default/files/wysiwyg/lhs/lhs_case_studies_hca.pdf
Agency for Healthcare Research and Quality,. University of Utah Health: Creating a Formula for Value-Based Care [Internet]. 2019 p. 7. Available from: https://www.ahrq.gov/sites/default/files/wysiwyg/lhs/lhs_case_studies_utah_health.pdf
Britto MT, Fuller SC, Kaplan HC, Kotagal U, Lannon C, Margolis PA, et al. Using a network organisational architecture to support the development of Learning Healthcare Systems. BMJ Qual Saf. 2018;27(11):937–46.
Forrest CB, Margolis P, Seid M, Colletti RB. PEDSnet: how a prototype pediatric learning health system is being expanded into a national network. Health Aff Proj Hope. 2014;33(7):1171–7.
Porcaro K. Building responsive governance for learning networks. Learn Health Syst. 2022;6(2): e10288.
Fung-Kee-Fung M, Maziak DE, Pantarotto JR, Smylie J, Taylor L, Timlin T, et al. Regional process redesign of lung cancer care: a learning health system pilot project. Curr Oncol Tor Ont. 2018;25(1):59–66.
Heys M, Kesler E, Sassoon Y, Wilson E, Fitzgerald F, Gannon H, et al. Development and implementation experience of a learning healthcare system for facility based newborn care in low resource settings: the Neotree. Learn Health Syst. e10310.
Lim SN, Keung C, Zhao L, Curcin V, Ethier JF, Burgun A, et al. Transform: Implementing a Learning Healthcare System in Europe through Embedding Clinical Research into Clinical Practice. In: 2015 48th Hawaii International Conference on System Sciences. 2015. p. 3176–85.
Koscielniak N, Jenkins D, Hassani S, Buckon C, Tucker JS, Sienko S, et al. The SHOnet learning health system: Infrastructure for continuous learning in pediatric rehabilitation. Learn Health Syst. 2022;6(3): e10305.
Levin A, Malbeuf M, Hoens AM, Carlsten C, Ryerson CJ, Cau A, et al. Creating a provincial post COVID-19 interdisciplinary clinical care network as a learning health system during the pandemic: Integrating clinical care and research. Learn Health Syst. e10316.
Moffatt-Bruce S, Huerta T, Gaughan A, McAlearney AS. IDEA4PS: the development of a research-oriented learning healthcare system. Am J Med Qual Off J Am Coll Med Qual. 2018;33(4):420–5.
Murray DS, Anixt SJ, Coury DL, Kuhlthau KA, Seide J, Kelly A, et al. Transforming an autism pediatric research network into a learning health system: lessons learned. Pediatr Qual Saf. 2019;4(2):e152.
Nash DM, Rayner J, Bhatti S, Zagar L, Zwarenstein M. The Alliance for Healthier Communities’ journey to a learning health system in primary care. Learn Health Syst. 2022;e10321:9.
Perito ER, Squires JE, Bray D, Bucuvalas J, Krise-Confair C, Eisenberg E, et al. A learning health system for pediatric liver transplant: the starzl network for excellence in pediatric transplantation. J Pediatr Gastroenterol Nutr. 2021;72(3):417–24.
Satveit S. Addressing the unique healthcare needs of women: opportunity for change exists at the intersection of precision health and learning health systems. Learn Health Syst. 2017;2(2): e10033.
Sinnige A, Spruijt S, Saes M, Van der Wees PJ, Hoogeboom TJ, Teijink JAW. Using a learning health system to improve physical therapy care for patients with intermittent claudication: lessons learned from the claudicationet quality system. Phys Ther. 2022;102(1):pzab249.
Taylor YJ, Kowalkowski M, Spencer MD, Evans SM, Hall MN, Rissmiller S, et al. Realizing a learning health system through process, rigor and culture change. Healthcare. 2021;1(8): 100478.
Foley T, Fairmichael F. The Learning Healthcare Project. 2015. Site visit to Geisinger Health System. https://learninghealthcareproject.org/site-visit-to-geisinger-health-system/. Accessed 21 Jul 2022.
Vandenberg AE, Kegler M, Hastings SN, Hwang U, Wu D, Stevens MB, et al. Sequential implementation of the EQUIPPED geriatric medication safety program as a learning health system. Int J Qual Health Care J Int Soc Qual Health Care. 2020;32(7):470–6.
Varnell CD, Margolis P, Goebel J, Hooper DK. The learning health system for pediatric nephrology: building better systems to improve health. Pediatr Nephrol Berl Ger. 2022;20:1–12.
Krapohl GL, Hemmila MR, Hendren S, Bishop K, Rogers R, Rocker C, et al. Building, scaling, and sustaining a learning health system for surgical quality improvement: a toolkit. Learn Health Syst. 2020;4(3): e10215.
Guise JM, Savitz LA, Friedman CP. Mind the gap: putting evidence into practice in the era of learning health systems. J Gen Intern Med. 2018;33(12):2237–9.
Stewart M, Brown JB, Donner A, McWhinney IR, Oates J, Weston WW, et al. The impact of patient-centered care on outcomes. J Fam Pract. 2000;49(9):796–804.
Lessard L, Michalowski W, Fung-Kee-Fung M, Jones L, Grudniewicz A. Architectural frameworks: defining the structures for implementing learning health systems. Implement Sci. 2017;12(1):78.
Jones RD, Krenz C, Gornick M, Griffith KA, Spence R, Bradbury AR, et al. Patient preferences regarding informed consent models for participation in a learning health care system for oncology. JCO Oncol Pract. 2020;16(9):e977–90.
Kelley M, James C, Alessi S, Korngiebel D, Wijangco I, Rosenthal E, et al. Patient perspectives on the learning health system: the importance of trust and shared decision making. Am J Bioeth AJOB. 2015;15(9):4–17.
Platt JE, Raj M, Wienroth M. An analysis of the learning health system in its first decade in practice: scoping review. J Med Internet Res. 2020;22(3): e17026.
Morain SR, Kass NE. Ethics issues arising in the transition to learning health care systems: results from interviews with leaders from 25 health systems. EGEMS Wash DC. 2016;4(2):1212.
Bubela T, Genuis SK, Janjua NZ, Krajden M, Mittmann N, Podolak K, et al. Medical information commons to support learning healthcare systems: examples from Canada. J Law Med Ethics. 2019;47(1):97–105.
Vinson AH. Culture as infrastructure in learning health systems. Learn Health Syst. 2021;5(3): e10267.
Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10(1):109.
Ellis LA, Sarkies M, Churruca K, Dammery G, Meulenbroeks I, Smith CL, et al. The science of learning health systems: scoping review of empirical research. JMIR Med Inform. 2022;10(2): e34907.
Boes S, Mantwill S, Kaufmann C, Brach M, Bickenbach J, Rubinelli S, et al. Swiss Learning Health System: a national initiative to establish learning cycles for continuous health system improvement. Learn Health Syst. 2018;2(3): e10059.
Acknowledgements
The authors would like to acknowledge Dr. Melissa Rothfus and Kristy Hancock for their support in designing, running and peer reviewing the search strategy for the review.
Funding
This work was supported by IWK Health Foundation’s Translating Research into Care Healthcare Improvement Research Funding Program [#1027492, 2021]. The lead author was supported by a CIHR-funded Health System Impact Fellowship.
Author information
Authors and Affiliations
Contributions
MS, CC, JC, DS, and AER designed the review protocol. MS, CC, CJ performed the title and abstract and full-text screening. MS and CJ performed data extraction, with CC as the third reviewer. MS data analysis, and the full team participated in data interpretation. MS wrote the first draft of the manuscript with CC and JC guidance. All authors critically reviewed and revised the manuscript and approved the final version.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Additional file 1: Table S1.
Example search strategy for online database, CINAHL (EBSCO), conducted July 28, 2022. Table S2. Detailed grey literature search strategy, conducted July 20, 2022. Table S3. JBI critical appraisal tool checklist questions. Table S4. Detailed overview of learning health system characteristics.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Somerville, M., Cassidy, C., Curran, J.A. et al. Implementation strategies and outcome measures for advancing learning health systems: a mixed methods systematic review. Health Res Policy Sys 21, 120 (2023). https://doi.org/10.1186/s12961-023-01071-w
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s12961-023-01071-w