FormalPara Key Summary Points

Traditional efficacy double-blind randomised controlled trials (DBRCTs) are considered the “gold standard” study design for assessing the efficacy and safety of new medicines; however, their conduct in highly selected patient populations and in highly controlled settings limits the generalisability of their findings to patients seen in everyday clinical practice.

Pragmatic effectiveness trials conducted in the routine clinical care setting allow for the evaluation of the effectiveness of medicines in the presence of real-world factors related to patients, actual medication use, and healthcare systems, thus providing a more complete picture of the benefit/risk profile of a medicine to support healthcare decision-making.

In this article, we discuss the key features and advantages/limitations of pragmatic effectiveness randomised controlled trials (RCTs) compared with traditional efficacy DBRCTs, using the Salford Lung Study (SLS) programme as an illustrative example.

SLS was the world’s first prospective, phase III, pragmatic RCT to evaluate the effectiveness of a pre-licensed medication in a primary care setting using electronic health records and through collaboratively engaging general practitioners and community pharmacists in clinical research.

Key learnings from SLS that may help inform the design of future pragmatic effectiveness RCTs include: (1) ensuring that the trial setting and operational infrastructure are aligned with routine clinical care; (2) recruiting a broad population of patients with characteristics as close as possible to patients seen in routine clinical practice, to maximise the generalisability and applicability of the trial results; (3) ensuring that patients and local healthcare professionals (HCPs) are suitably engaged in the trial, to maximise the chances of successful trial delivery; and (4) careful study design, incorporating outcomes of value to patients, HCPs, policymakers and payers, and using pre-planned analyses to address scientifically valid research hypotheses to ensure robustness of the acquired data.

Introduction

Double-blind randomised controlled trials (DBRCTs) are considered the “gold standard” study design for assessing the efficacy and safety of new medicines, and are designed to achieve maximum internal validity with minimal potential for confounding factors [1]. Frequently conducted for the purpose of obtaining data to support regulatory approvals, DBRCTs underpin the evidence base informing treatment guidelines and healthcare decisions [1,2,3]. However, as efficacy DBRCTs are conducted in highly selected patient populations and under highly controlled conditions (Fig. 1) optimised to demonstrate the effect of the medicine, the generalisability of their findings to the overall disease population may be limited [3,4,5,6,7].

Fig. 1
figure 1

Design aspects of traditional efficacy DBRCTs compared with pragmatic effectiveness RCTs. DBRCT double-blind randomised controlled trial, RCT randomised controlled trial

Full size image

Pragmatic randomised effectiveness trials are designed to evaluate medicines in the “real-world” setting across a broad patient population [8, 9] (Fig. 1) and offer the opportunity to address issues faced by patients and healthcare professionals (HCPs) on a daily basis [3, 6], while retaining the benefits of random treatment allocation. Randomised effectiveness trials can complement traditional efficacy DBRCTs by filling the evidence gaps surrounding patient and physician experience, treatment adherence, and healthcare resource utilisation (HRU) and care costs, all of which are key to informing healthcare decision-making [9].

The phase IIIb, pragmatic effectiveness Salford Lung Study (SLS) RCT programme was designed to evaluate a pre-licensed inhaled corticosteroid/long-acting β2-agonist combination, fluticasone furoate/vilanterol (FF/VI), in patients with chronic obstructive pulmonary disease (COPD) and asthma in UK primary care [10]. SLS was a world first, embracing the novel use of electronic health records (EHRs) to comprehensively enrol a broad spectrum of patients from across Salford and surrounding areas. Here, we discuss key features and advantages/limitations of pragmatic effectiveness RCTs versus traditional efficacy DBRCTs (focusing on respiratory trials), using SLS as an illustrative example. We also describe key learnings from SLS and discuss how these might help inform the design of future pragmatic effectiveness RCTs and respiratory treatment guidelines/healthcare policies.

Efficacy and Effectiveness RCTS: Overview and Major Differences

Table 1 compares features of traditional efficacy DBRCTs versus pragmatic effectiveness RCTs. The strict inclusion/exclusion criteria of traditional efficacy DBRCTs are designed to maximise internal validity and reduce the impact of biases. However, in the real-world setting, patients with COPD and asthma arrive at the doctor’s office with many confounding/complicating factors not assessed in DBRCTs, which can have profound effects on the likelihood of a medicine causing benefit or harm (Box 1).

Table 1 Comparison of traditional efficacy DBRCTs and pragmatic effectiveness RCTs
Full size table
Box 1 Examples of patient confounding/complicating factors in the real-world clinical practice setting
Full size table

Almost all of these factors are excluded/altered by the strict eligibility criteria, conduct, and need for protocol compliance in traditional efficacy DBRCTs. Consequently, only a low proportion of primary care patients with COPD and asthma would be eligible for participation in typical efficacy trials [4, 11,12,13,14] and the relevance of DBRCT results to patients in routine practice is limited [1, 7]. Effectiveness RCTs can therefore supplement data from efficacy DBRCTs by providing a more complete picture of the benefit/risk profile of a medicine to support healthcare decision-making.

The inherent design differences between traditional efficacy DBRCTs and pragmatic effectiveness RCTs result in different strengths and limitations of the acquired data (Box 2).

Box 2 Traditional efficacy DBRCTs versus pragmatic effectiveness RCTs: impact of study design aspects on acquired data
Full size table

The Salford Lung Studies in COPD and Asthma: What Were Their Novel “Real-World” Aspects?

The SLS programme comprised two concurrent, 12-month, open-label, phase IIIb pragmatic RCTs designed to evaluate the effectiveness and safety of initiating once-daily inhaled FF/VI, compared with continuing usual care (UC) in patients with COPD or asthma in UK primary care [10, 15,16,17,18]. All patients provided written informed consent for participation in SLS and the trials were conducted in accordance with the International Conference on Harmonisation Good Clinical Practice guidelines and the provisions of the 2008 Declaration of Helsinki. The trial protocols were approved by the National Research Ethics Service Committee North West, Greater Manchester South (approval numbers 11/NW/0798 and 12/NW/0455).

When SLS commenced, a full regulatory submission for FF/VI was under consideration by the European Medicines Agency that was based on extensive efficacy and safety data from completed RCTs [10]. SLS was conducted to meet the need for effectiveness data for FF/VI to complement existing evidence from efficacy RCTs. SLS was the first study in the UK to take advantage of joint advice from the Medicines and Healthcare products Regulatory Agency (MHRA) and The National Institute for Health and Care Excellence (NICE). MHRA, responding positively to NICE’s enthusiasm for pragmatic data in a broad community, approved the study design with a pre-licensed medicine—a key factor enabling the study to proceed.

SLS evaluated the effectiveness of a pre-licensed medication in the real-world setting using EHRs [10, 19] and collaboratively engaged general practitioners (GPs) and community pharmacists in clinical research. The SLS trial designs have been reported previously [10, 15,16,17,18] (Fig. 2). The studies employed broad eligibility criteria to recruit large, heterogeneous populations of patients with COPD and asthma. There were few protocol-mandated clinic visits and data were collected continuously and remotely from patients’ EHRs using a primary/secondary care-linked database system (as well as via electronic case report forms, eCRFs). Patients were recruited and managed by their usual GPs, who prescribed as normal, and patients ordered and collected repeat prescriptions in the usual way and collected their study medication from their usual community pharmacist. Treatment modifications were permitted at GPs’ discretion during the study; patients randomised to initiate FF/VI could modify their treatment to any other appropriate treatment and remain in the FF/VI randomisation group. Those randomised to continue UC were also allowed to modify their treatment to any other appropriate treatment (except for FF/VI) and remain in the UC randomisation group. The real-world design aspect of SLS was unique, requiring careful planning and attention to the goals of maximising the external validity of the trials while maintaining scientific rigour, as well as securing suitable electronic processes for proper interpretation of safety data.

Fig. 2
figure 2

SLS trial designs. a SLS COPD, b SLS asthma. ACT Asthma Control Test, COPD chronic obstructive pulmonary disease, FF/VI fluticasone furoate/vilanterol, GP general practitioner, ICS inhaled corticosteroid, LABA long-acting β2-agonist, LAMA long-acting muscarinic antagonist, SLS Salford Lung Study, UC usual care, y years

Full size image

Maximising External Validity

Careful design and much background work went into ensuring that the delivery of SLS was aligned with routine care (e.g. normality of medicines supply; patient and HCP behaviours consistent with everyday clinical practice; interplay between patients, GPs, pharmacists in the community setting), while ensuring that the study conduct and data collection met the requirements of a phase IIIb RCT.

Collecting patients’ data via EHRs allowed us to measure the SLS COPD primary effectiveness endpoint (moderate/severe exacerbations) through surrogates triggered in the EHR (prescription of antibiotics and/or systemic corticosteroids or hospital admissions/visits associated with a respiratory cause [17]). This ensured that patients and doctors were essentially unaffected by the study for the entire follow-up period. In SLS asthma, the same primary effectiveness endpoint was not feasible because of the expected low frequency of exacerbations based on a pilot study (recruitment numbers would have been too high to achieve the required statistical power); instead, response on Asthma Control Test (ACT) was selected as the primary outcome. ACT was completed by patients at the baseline (randomisation) and week 52 (end of study) scheduled visits, and was additionally administered via telephone at weeks 12, 24, and 40 [18]. Processes were implemented to ensure that ACT was administered with minimal interference to normal care (e.g. GPs were aware of ACT scores at baseline, but not thereafter; telephone ACT was administered by a study nurse blinded to treatment and who was trained not to provide advice to the patient, except under life-threatening circumstances).

All HCPs involved in SLS (GPs, nurses, pharmacists, and their staff) were trained to allow routine clinical practice to proceed, although consultation rates were higher in the FF/VI randomisation group than the UC group during the first 12 weeks [20, 21], predominantly for non-respiratory reasons, suggesting that GPs and patients did undergo an initial familiarisation period with the new therapy. Importantly, by extracting HRU data directly from EHRs, we were able to obtain a complete picture of the burden associated with COPD and asthma without bias due to recall or transcription of data, and were able to demonstrate a disproportionately high resource use for non-COPD/-asthma reasons [20, 21].

This “hands-off” approach really allowed normal patient and HCP behaviours to play out in SLS—quite unusual for a phase III trial, and a very positive aspect of the trial design, as it provides valuable information about how FF/VI performs when used in routine practice.

Maintaining Scientific Rigour

Through its prospective design, baseline randomisation/stratification procedures, and extensive a priori analysis plan, SLS achieved the scientific rigour characteristic of a traditional efficacy RCT. Much consideration went into the decision to allow asymmetric treatment modification in the trial design and the subsequent impact this would have on the data analyses (analysis by randomised treatment group or by actual treatment).

Furthermore, for the purpose of statistics and programming, the sponsor remained blinded to study treatment while the trial was ongoing and was only unblinded after all data had been collected and the study database had been locked, thus mimicking the approach taken in a typical DBRCT.

Safety Data Collection

SLS commenced with a pre-licensed medicine and our intention was to vigilantly collect and evaluate safety events in real time through patients’ EHRs. In recruiting a population inclusive of patients with comorbidities and severe disease, we anticipated that the study would accrue a large volume of safety data and that patients would experience multiple serious adverse events (SAEs) that would not be seen in trials where such comorbidities are excluded. An innovative approach for identifying potential AEs without interfering with patients’ normal routines was essential. The SLS safety data collection process has been published previously [19], but a key aspect of this was the creation of a consultant-led specialist safety team, who were alerted to review potential safety events in real time.

Over 7000 patients participated in SLS and both trials met their primary effectiveness endpoints, demonstrating the benefit of initiating FF/VI versus continuing UC [17, 18]. In SLS COPD, there was a statistically significant reduction for FF/VI versus UC in the mean annual rate of moderate/severe exacerbations, without increased risk of SAEs [17]. In SLS asthma, the odds of patients being ACT responders (ACT total score ≥ 20 and/or improvement from baseline ≥ 3) at week 24 were significantly higher for FF/VI versus UC, without increased risk of SAEs [18]. Consistent benefit of FF/VI over UC has also been demonstrated for various other endpoints, as demonstrated in secondary analyses of SLS COPD and asthma [20,21,22,23,24,25,26,27,28,29].

SLS Approach to Effectiveness RCTS: Advantages and Limitations

Conducting effectiveness RCTs such as SLS in routine primary care requires access to the patient population of interest and good infrastructure, operational management, training/good clinical practice, and site engagement. It could be argued that such requirements could preclude the conduct of similar effectiveness studies in other geographical locations [30].

As SLS comprised phase IIIb trials of a pre-licensed medication requiring detailed safety monitoring, the studies were time- and resource-intensive to design and the operational logistics were highly complex. There exists a perception that effectiveness studies are simpler to design and less expensive to implement than traditional efficacy RCTs, but our experience with SLS suggests quite the opposite; however, this may not necessarily reflect requirements for all real-world effectiveness studies, which should be designed on a case-by-case basis.

SLS commenced with a phase of pre-licensed FF/VI in the UK primary care setting and this had implications for the acquired trial data set, which should be considered when designing similar future effectiveness RCTs. For example, in SLS COPD, higher rates of all-cause primary care contacts for FF/VI versus UC were observed in the first 3 months of the study, which may have been driven by additional scrutiny of the then-unlicensed FF/VI [20]. Patients and physicians were allowed to modify treatment, and this required an additional level of consideration around the underlying reasons for modification and had implications for data analysis. In SLS, we were able to determine actual HRU and care costs (as opposed to the usual modelled costs) for patients with COPD and asthma, which is highly relevant for routine clinical practice in the UK. Furthermore, patients from deprived areas, who may be ineligible for, or unwilling to participate in, traditional phase III RCTs were recruited to the SLS. Salford, UK, is an urban location with areas of deprivation. Asthma patients were equally represented in deprivation categories, but COPD patients are over-represented in deprived areas. In an analysis of outcomes by deprivation, we found that deprivation did not impact the main outcomes of the SLS trials, thus supporting the recruitment of participants from all socioeconomic strata to provide data that are generalisable to routine clinical practice [26,27,28].

To reflect the routine clinical care setting, the UC randomisation group comprised many different inhaled maintenance therapies as the comparator for initiation of FF/VI, which could also be varied over the course of the study. Caveats of this design aspect include a limited capacity to prospectively evaluate FF/VI against a specific UC treatment, and inability to equate a UC option with an established standard-of-care in COPD or asthma. Allowing broad UC therapies and treatment modifications in SLS has limitations for data analysis (precludes direct, head-to-head comparisons of FF/VI versus UC, or of FF/VI versus one particular UC treatment), but advantages in that analysis by actual treatment could be conducted (e.g. safety reporting).

How Might SLS Inform HCPS and Decision-Makers?

For most newly approved medicines, evidence from efficacy DBRCTs is insufficient to fully guide physicians in choosing optimal treatment for their patients. Pragmatic effectiveness trials can fill the gap by providing data on the overall treatment strategy in routine clinical practice while maintaining the strength of an RCT [3] and are a valid option for addressing issues that patients, clinicians and policymakers face on a daily basis. Knowledge of the overall effectiveness of a medicine in the intended patient population, taking into account real-world factors related to patients, actual medication use, and healthcare systems, will ultimately help HCPs make more-informed treatment choices for their patients. Furthermore, this study is the first to provide HCPs with answers on how initiating FF/VI (having been treated previously with other medicines) may impact exacerbation rates and other outcomes versus continuing on those other medicines.

In respiratory clinical care, there has tended to be a focus on symptom management; however, patients are often more concerned with how their symptoms make them feel and the impact of symptoms on their everyday lives [31]. Health care is increasingly adopting a patient-centric approach, which considers patients’ perspectives regarding the impact of disease and its treatment. Clinical trials should, therefore, incorporate appropriate patient-reported outcomes (PROs) and quality of life (QoL) outcomes in their design. In SLS asthma, several PRO effectiveness endpoints were prospectively assessed (including ACT, Asthma Quality of Life Questionnaire, Work Productivity and Impairment Questionnaire: Asthma, and the EuroQol 5-Dimensions 3-Levels questionnaire). Initiation of FF/VI was associated with consistent benefits in PROs versus continuing UC, demonstrating that the observed improvement in asthma control (measured by ACT) for FF/VI translates into patient-perceived benefits in health-related QoL [23]. Follow-up interview-based studies conducted in subsets of patients who completed SLS have also provided important additional findings on patient-centred outcomes relevant to respiratory care in routine clinical practice [24, 32, 33].

For healthcare policy decision-makers, data from effectiveness trials can provide a more balanced view of the overall benefit/risk of a medicine, including HRU and cost-effectiveness—critical factors for consideration by resource-limited health services [10]. However, despite Health Technology Assessment groups expressing a desire to see more pragmatic studies describing effectiveness, for many, their dossier restrictions do not allow unblinded studies to be included in their assessments. There have been few studies like SLS in phase III and so payers and regulators have little experience of the nuances of such data sets and how to respond to them within their regulation. However, unless they do so, this will have a negative impact on sponsors’ willingness to fund effectiveness studies.

Learnings From SLS: How Can These Be Applied to Future RCTS?

Pragmatic real-world study design requires careful consideration of the setting, patient population, intervention, comparator, and outcomes [3]. Here, we discuss key learnings from SLS and how these might help inform the design of future effectiveness RCTs. Considerable effort and time was spent on aspects of the study design and operationalisation beyond our experience in DBRCTs; these aspects are summarised in Table 2.

Table 2 SLS design aspects—issues requiring greater effort/input to resolve compared to DBRCTs, based on opinion of SLS investigators and operational staff
Full size table

Setting and Infrastructure

The underlying operational infrastructure was the key to delivering two large phase IIIb trials evaluating a pre-licensed medicine out of local GP practices and pharmacies in and around Salford, UK. Salford was an ideal location for the trials owing to its relatively static population served by a single hospital with an integrated, real-time EHR connection with surrounding primary care practices and linking with patient-level prescription information (Salford Integrated Record, SIR) [10]. A bespoke information technology infrastructure was developed by NorthWest eHealth to extract data from the SIR for the purpose of effectiveness research [10, 19]. Over 2100 GPs, nurses, and pharmacy staff in and around Salford were trained in good clinical practice in SLS.

Critical to the delivery of SLS was the unique involvement and collaboration of community pharmacies. A key component of the integrity of the effectiveness design was maintaining the normality of repeat prescribing and dispensing in a situation where a pre-licensed medicine was being evaluated. Extensive training and process development permitted pharmacists/pharmacy staff to participate in a phase III trial, despite most having no prior clinical research experience. SLS demonstrated that pharmacies normally competing for prescription business can work in a collaborative and supportive manner for the benefits of patients. Specific aspects of how pharmacy involvement was successfully achieved in the SLS will be the subject of another publication [34].

Patient Population: Inclusiveness and Applicability

SLS showed that by employing limited eligibility criteria, it is possible to recruit to a phase IIIb effectiveness RCT a broad population of patients that are representative of those in everyday clinical practice, including from socioeconomically deprived areas.

SLS COPD enrolled approximately half of all eligible patients with COPD in the target geographical area [7]. These patients had a high disease burden and more symptoms, more frequent exacerbations, more comorbidities, and more SAEs/pneumonia SAEs compared with patients in large, registrational, efficacy RCTs in COPD [7]. Furthermore, over half of SLS COPD patients were categorised in the most deprived quintile by postcode [26, 28]. Notably, only at most 30% of SLS COPD patients would have been eligible for a typical regulatory phase III COPD exacerbation study [7], definitively demonstrating that COPD patients enrolled in traditional efficacy RCTs are not representative of patients in primary care.

The applicability of SLS findings to patients in routine clinical care is supported by a previous study demonstrating similarity in the characteristics of SLS COPD patients with other primary care patient populations across Europe [35]. Further support comes from a recent observational study demonstrating that patients in the SLS COPD UC group were similar to a matched cohort of non-trial COPD patients in England from the Clinical Practice Research Datalink database [36]. Evidence for a Hawthorne effect was observed, with a higher frequency of COPD exacerbations recorded in SLS patients than in non-trial patients; however, the largest effect was observed through behavioural changes in patients and GP coding practices [36]. SLS data have, therefore, contributed to the development of novel methods for evaluating the presence of an operating Hawthorne effect for future effectiveness trials conducted in everyday clinical practice.

Patient and HCP Engagement

Our experience with SLS underscores the importance of carefully designing pragmatic effectiveness RCTs to maximise chances of success in routine practice while ensuring operational feasibility. Engaging patients and HCPs in effectiveness research is extremely challenging. Initially, patient enrolment was slow in the SLS and we had to revisit our strategies for recruitment. The key was the involvement of patients’ own GPs in recruitment and obtaining of consent. Following on from SLS, additional qualitative research has been conducted to understand the drivers for patient and HCP engagement in the studies and how participation in future effectiveness trials might be enhanced [37]. Though key learnings from SLS will be the subject of a separate publication, our findings should be highlighted around the overall positive experience of patients and HCPs who participated in SLS and the importance of (1) local advertising to raise community awareness of study recruitment; (2) site/investigator engagement, ensuring that through extensive training on Good Clinical Practice, the study design and delivery was aligned with routine care; (3) provision of research nurse support at study sites, which was key to study delivery; (4) ease and convenience of study assessments; and (5) a need for improved study results dissemination [37].

Outcomes

SLS incorporated outcomes of interest outside those typically included in traditional respiratory efficacy DBRCTs (e.g. HRU/care costs, PROs, patient exit interviews), which has provided valuable supporting data for the benefit of initiating FF/VI versus continuing UC in patients with COPD and asthma in routine practice [20,21,22,23,24, 32, 33].

Furthermore, allowing asymmetric treatment modifications in the study design necessitated a novel approach to safety evaluation: highly comprehensive safety analyses were conducted both by randomised treatment group and by actual treatment at the time of an event.

The scientifically rigorous collection of real-world data in SLS offers major opportunities for future studies examining new research questions.

Conclusions

The real-world design aspect of SLS was unique, requiring careful planning and attention to the goals of maximising external validity while maintaining scientific rigour and securing suitable electronic processes for safety data collection. Key learnings from SLS that may help inform the design of future pragmatic effectiveness RCTs include: (1) ensuring that the trial setting and operational infrastructure are aligned with routine clinical care; (2) recruiting a broad population of patients with characteristics as close as possible to patients seen in routine clinical practice, to maximise the generalisability and applicability of the trial results; (3) ensuring that patients and HCPs are suitably engaged in the trial, to maximise the chances of successful trial delivery; (4) careful study design, incorporating outcomes of value to patients, HCPs, policymakers and payers; and (5) using pre-planned analyses to address scientifically valid research hypotheses to ensure robustness of the acquired data.