FormalPara Key Points for Decision Makers

myCOPD has the potential to help adults manage their chronic obstructive pulmonary disease (COPD); however, there is currently insufficient evidence to support the case for routine adoption in the NHS.

Research comparing myCOPD with standard care is recommended to address uncertainties about the claimed benefits of using myCOPD.

1 Introduction

The National Institute for Health and Care Excellence (NICE) evaluates medical technologies and produces evidence-based guidance to help improve health and social care in England. NICE’s Medical Technologies Evaluation Programme (MTEP) selects medical technologies for evaluation that have the potential to reduce costs to the healthcare system and offer clinical benefits to the patient compared with standard care.

A scoping document detailing the decision problem is produced by NICE, and clinical and economic evidence submitted by the company is assessed independently by an External Assessment Group (EAG). Following the EAG’s evaluation and public consultation period, the Medical Technologies Advisory Committee (MTAC) develops guidance [2, 3].

In 2019, NHS England commissioned NICE to develop a medical technologies guidance development process for digital health technologies (DHTs), which is being piloted on a selected number of topics. The proposed process is based on the NICE MTEP process but is adapted to consider some of the characteristics of digital technologies.

In March 2022, NICE issued final guidance on myCOPD for managing chronic obstructive pulmonary disease (COPD). myCOPD is a digital tool designed to enable people to manage their COPD. The platform is designed to allow shared decision making between patient and clinician to improve self-efficacy and help individuals to manage their COPD effectively with the support of myCOPD.

The EAG critiquing the evidence was York Health Economics Consortium. Clinical experts, identified using NICE’s published processes, provided advice to the EAG and MTAC.

This article includes an overview of the clinical and cost-effectiveness evidence submitted by the company, the EAG’s report, and subsequent development of the NICE guidance. Full documentation including final guidance and supporting evidence is available on the NICE website [1].

1.1 Background

COPD is a lung disease characterised by airflow limitation [4]. It can be caused by exposure to harmful substances, for example cigarette smoking, and is a common cause of death worldwide [4]. COPD is common in adults and treatment involves strategies to control symptoms, improve quality of life and reduce exacerbations and mortality [4]. Pharmacological treatments include bronchodilators, inhaled corticosteroids, systemic glucocorticoids, phosphodiesterase-4 inhibitors and antibiotics. Non-pharmacological treatment includes smoking cessation and pulmonary rehabilitation (PR) [4]. myCOPD (my mHealth) is an application (app) that can be installed onto a device such as a smartphone, tablet or computer via an internet connection. Licencing is available via different price packages. At the time of the assessment, an ‘unlimited licence package’ was available as part of a 3-year contract with a clinical commissioning group (CCG). myCOPD incorporates tools for patient education, self-management, symptom tracking and PR. Individuals can record their symptoms daily and they may periodically undertake a COPD assessment test (CAT). With the user’s permission, clinicians can access the results of these assessments and patient medication records, so that monitoring and management (for example, suggesting a change to inhaler prescriptions) can be undertaken remotely.

The PR element of the tool is a 6-week online course comprising incremental exercise training and education sessions promoting effective self-management of COPD, and is conducted remotely.

2 Decision Problem (Scope)

2.1 Population

The population described in the scope included people with a diagnosis of COPD. The company did not propose any variation to the scope but in the evidence submission, the company focused on two subgroups: a cohort who had been discharged from hospital for acute exacerbation of COPD (AECOPD) and a cohort with stable COPD eligible for PR. It is likely that there is a small overlap between these populations.

2.2 Intervention and Comparator

The intervention identified in the scope was myCOPD as an add-on intervention to standard care, and the comparator listed in the scope was standard care without myCOPD.

All of the clinical studies identified by the EAG assessed myCOPD in addition to standard care and compared it with standard care alone; however, the definition of standard care varied across studies.

2.3 Outcomes

The scope included 11 outcomes. The company provided evidence for all of these, including COPD symptom assessment CAT score, rates of acute exacerbation and hospital admissions, number of healthcare professional (HCP) consultations (limited real-world evidence only), rates of inhaler error, adherence to the use of myCOPD, health-related quality of life (HRQoL), patient activation measures, self-efficacy for appropriate medication use, a walking test and device-related adverse events.

3 Review of Clinical and Economic Evidence

3.1 Clinical Effectiveness Evidence Identification and Selection

The company submitted four published peer-reviewed clinical studies, including three randomised controlled trials (RCTs) [5,6,7] and one observational study [8]. One study was ongoing at the time of its submission [9]. The company also submitted six published and nine unpublished real-world evaluations (RWEs) of myCOPD conducted in seven settings (Ipswich and East Suffolk [10], Southend [11], Leeds [12], Coventry [13], Grampian [14], Highland [15] and West Lothian [16]). The company shared usage information (as of January 2021) [17] and unpublished responses from six clinicians to a company questionnaire about the usefulness of the app and whether myCOPD could be incorporated into the clinical pathway [18].

The company did not provide information on the selection criteria or search strategy to identify relevant studies in its submission report. It was therefore not possible to assess whether the search methodology was appropriate, and hence the EAG conducted a de novo literature search (see the electronic supplementary material [ESM]) to identify evidence.

The EAG searched a range of resources containing details of published, unpublished and ongoing research. The search was originally conducted in October 2019, then updated in January 2021 after it was selected for the guidance development. From the 2019 and 2021 searches, 7761 records in total were retrieved, with 3280 remaining after deduplication for assessment.

The EAG’s search did not identify any clinical studies that were not stated in the company’s submission, but a further five published RWE papers were identified by the EAG’s searches, including an additional three RWE settings (Kent [19], Mid and South Essex [20] and Dorset [21]).

No meta-analysis was conducted by the company or by the EAG.

3.2 Critique of Clinical Effectiveness Evidence

Evidence on effectiveness was available for the two myCOPD applications: self-management and PR. For self-management, two RCTs (RESCUE [7] and EARLY [6]), one observational study [8] and a number of RWE reports provided evidence relating to the use of myCOPD. For PR, one RCT (TROOPER) [5] and a number of RWE reports provided evidence relating to the use of myCOPD. For the quality assessment of studies, the EAG used the criteria proposed by the Centre for Reviews and Dissemination for RCTs (Khan et al., 2001) and the Critical Appraisal Skills Programme (CASP) checklist for cohort studies for the comparative observational studies (CASP UK 2013).

The company did not conduct a critical appraisal of the included studies and the EAG undertook its own critical appraisal for each of the clinical studies. The internal and external validity of the RCTs were considered to be acceptable. For the observational study [8], external validity was judged to be acceptable but internal validity was judged to be low because of limited reporting of intervention details and outcome measures and no assessment of confounding factors. The EAG did not conduct quality assessment of real-world evidence because of the unsubstantive nature of outcomes and variation in the methods in these studies. The EAG considered these studies to have acceptable external validity but low internal validity.

3.2.1 Self-Management

The two RCTs evaluating myCOPD for self-management of COPD were RESCUE (compared myCOPD with ‘usual care with additional written support’) [7] and EARLY (compared myCOPD with 'usual care') [6]. Both trials were small, including 41 and 60 participants respectively. Details of power calculations were reported in EARLY. In RESCUE, the authors noted that the study was small and had limited power to demonstrate the effects on all measured outcomes. However, the authors considered that the sample size was suitable for the main objective of their pilot study—to determine the feasibility of using a digital platform to support participants with COPD after a significant clinical event such as exacerbation. Outcomes were measured at baseline and at three months in both trials. The trials did not have any longer-term follow up.

In terms of outcome for the RCTs, although there was a tendency for myCOPD to improve the CAT score in the RCTs, findings were not statistically significant and positive effects were not consistently shown for other outcomes (acute exacerbations, inhaler errors and HRQoL).

The observational study [8] explored the efficacy of myCOPD use compared with standard care without using myCOPD. The study matched the scope of the decision problem in terms of its populations; it was small (n = 36) and was reported as a brief article with limited details of the study methods.

RWE focused on user acceptance and adherence, and many were pilot studies using the results to inform decisions on whether to commission the app more widely.

3.2.2 Pulmonary Rehabilitation

The RCT of myCOPD for PR, TROOPER [5], is a non-inferiority RCT that compared myCOPD PR with a ‘face-to-face’ PR programme. There were 90 participants in the trial and although details of the power calculation were reported, there was uncertainty regarding its validity. Outcomes were measured at baseline and one week after completion of a six-week PR programme in TROOPER with no longer-term follow up.

The TROOPER trial found that myCOPD was non-inferior compared with face-to-face PR for CAT score (adjusted mean difference −1.0, 95% confidence interval [CI] −2.9 to 0.86; p = 0.373) and six-minute walking test (adjusted mean difference 23.8 m, 95% CI −4.5 to 52.2; p = 0.098), and there was an indication of non-inferiority for other measures, e.g. anxiety and quality of life. Rates of acute exacerbation and hospital admissions were not reported; however, due to the small sample size in this trial, caution was noted in the interpretation of findings.

The RWE for PR focused on user acceptance and adherence and many were pilot studies using the results to inform decisions on whether to commission the app more widely.

3.2.3 Clinical Effectiveness Summary

Overall, the evidence was considered uncertain due to the non-significant findings in the RCTs of self-management and the lack of power in the RCT of PR. Positive findings from RWE suggested the potential for real-world implementation but these studies are mainly interim service evaluations that do not provide robust evidence of effectiveness. App usage fell over time in all three RCTs and in the RWE, and there was some question over the long-term effectiveness.

3.3 Economic Evidence

The economic evidence was assessed in July 2021. The company did not include any economic evidence in its submission. The EAG rerun the searches based on the company search methods, as well as conducting a de novo economic evidence literature search (see the ESM). No economic studies suitable for inclusion were identified by the EAG.

The company submitted two de novo cost models comparing myCOPD with standard care. The company focused their models on subgroups of the COPD population in line with the clinical evidence.

3.3.1 Acute Exacerbation of Chronic Obstructive Pulmonary Disease Model (Self-Management)

The first subgroup modelled was a cohort who had been discharged from hospital for AECOPD. Standard care for this population was a written self-management plan at discharge. A cost calculator with a one-year time horizon was developed in Treeage. The model was based on a CCG purchasing the unlimited myCOPD licence package (myCOPD was covered for the whole CCG). Outcomes were modelled over three months for myCOPD and standard care, and included the rate of general practioner (GP) appointments, non-hospital admitted exacerbations and hospital readmissions for COPD (Table 1). Efficacy data for the model, including the rate of exacerbations and hospital readmissions, were sourced from the RESCUE RCT [7], and the number of GP appointments was sourced from the RWE by McLaughlin and Skinner [14] (see Table 1).

Table 1 Clinical parameters used in the company’s model and any changes made by the EAG [1]
Full size table

Base-case results from the company’s AECOPD model estimated that myCOPD generated cost savings of £204,641 per CCG. Best- and worst-case scenarios examined the impact of varying the model inputs on the results (best-case scenario: £1,785,878 cost saving per CCG; worst-case scenario: £69,530 cost increase per CCG; see ESM Table S3 for inputs). A tornado diagram showed that the readmission rate over 90 days post AECOPD was the key driver of results. The 90-day readmission rate in the myCOPD arm at which the base-case model became cost neutral/cost incurring was 0.357.

Overall, the EAG considered the company’s AECOPD model structure to be appropriate. However, the model outcomes were applied to every person who was discharged from hospital with AECOPD, meaning the uptake of myCOPD was assumed to be 100%. The EAG considered the 100% uptake rate to be optimistic and amended the uptake rate to 46% to account for the proportion of people who would not agree to be registered for myCOPD. This was based on data from an RCT [22]. The uptake rate was varied in a sensitivity analysis to account for any uncertainty in this value. No other changes were made to the model structure but existing inputs were updated (see Tables 1 and 2).

Table 2 Cost parameters used in the company’s model and any changes made by the EAG [1]
Full size table

The changes made to the AECOPD model by the EAG resulted in a reduced cost saving, from £204,641 to £86,297 per CCG, influenced mainly by the amended uptake in the model, although reducing the proportion of AECOPD patients who registered for myCOPD also reduced resource use costs.

In the EAG’s model, the best-case and worst-case scenario results were a cost saving of £4,143,428 per CCG (best-case scenario) and a cost increase of £58,928 per CCG (worst-case scenario). The EAG also identified the key driver from the sensitivity analysis as the hospital 90-day readmission rate. The point at which myCOPD changed from being cost saving to cost incurring was when the uptake rate was 26.2% or when the per person 90-day readmission rate was 0.30.

The EAG conducted probabilistic sensitivity analysis (PSA), which estimated that myCOPD had a 73.5% probability of being cost saving (see ESM Table S4 for sensitivity analysis inputs).

3.3.2 Pulmonary Rehabilitation Model

The second subgroup modelled was a cohort of patients eligible for PR with stable COPD, defined as Medical Research Council (MRC) grade three dyspnoea or above, who had been discharged from hospital post AECOPD. Standard care for this population was face-to-face PR (a six-week programme consisting of two supervised and three additional unsupervised sessions a week).

The company developed a decision tree in Treeage. Patients enter the model with a face-to-face PR assessment. Patients in the myCOPD arm can either be treated with face-to-face PR, a mixture of face-to-face sessions plus use of myCOPD, or use of myCOPD only. The PR element of the myCOPD app is intended to be used instead of standard care, deviating from the NICE scope, which specifies for myCOPD to be used in addition to standard care. Patients did not have to complete the full PR programme. Outcomes were modelled over a one-year time horizon and included the rate of exacerbations (see Table 1 for parameter inputs). The company presented two sets of results—incremental costs per CCG and incremental costs per PR provider. It must be noted that the former set of results cannot be interpreted as stand-alone because it was assumed that the myCOPD licencing is already in place.

Base-case results from the company’s PR model estimated that myCOPD generated cost savings of £20,269 per CCG; the alternative scenario presented a cost saving of £8707 per PR service provider. The company presented a tornado diagram to present the key drivers for the CCG costing model results (see ESM Table S5 for inputs). For the base case, the only parameter that changed the direction of the results was the number of referrals to PR when using myCOPD. For the PR service scenario, the key drivers identified by the company were the probability of being treated with myCOPD only and with a hybrid model (face-to-face and myCOPD), and the cost per face-to-face PR treatment. The company also presented a threshold analysis that identified that 276 referrals per year to PR were necessary for myCOPD to be cost saving.

Overall, the company’s PR model and structure were judged to be appropriate. However, the EAG concluded that a cost should be included for people who begin, but do not complete, the PR programme. The EAG included this, but any benefits of partially completing a PR programme were not captured in the number of exacerbations assigned to patients not completing a PR programme.

The EAG also changed the decision point in the model from referral to PR, to the point at which patients have expressed their willingness to use myCOPD. This was to better align the starting point of the two myCOPD models, to align the treatment arm with the scope (treatment options being myCOPD alone or hybrid option only, removing the option for only face-to-face), and to align the model with the population in the TROOPER study (the main efficacy data used in the model).

Changes made by the EAG to the PR model resulted in an increased cost saving, from £20,269 to £22,779 per CCG, or £8707 to £11,093 per PR service provider if considering the PR costing scenario.

Sensitivity analysis showed that for the base-case results, key drivers were the probability of being treated with myCOPD only, probability of being treated with the hybrid model, and the probability of referral to PR. In the base-case model, it is not possible for the results to become cost incurring unless the cost of registering a patient on the myCOPD app outweighs all of the other cost benefits. This is because the CCG is assumed to have already purchased myCOPD. In threshold analysis, myCOPD may not be cost saving in PR services with fewer than 240 referrals per year. Furthermore, the probability of being treated with myCOPD would need to be 1.9% when the hybrid model uptake is 12.2%, or 9.8% if the hybrid model is not used, for myCOPD to be cost saving. Therefore, if a hybrid model is not used, uptake of myCOPD would need to be higher to demonstrate a cost saving.

The EAG conducted PSA, which estimated myCOPD had an 86% probability of being cost saving in the CCG model and 87% probability of being cost saving in the PR service scenario.

3.3.3 Economic Effectiveness Summary

Due to no evidence of benefit in the whole population of people with COPD, the company modelled subgroups where benefit could be demonstrated (a cohort who had been discharged from hospital for AECOPD, and a cohort with stable COPD referred for PR).

The AECOPD model demonstrated that myCOPD was cost saving in this subpopulation, dependent on myCOPD uptake and hospital readmission rates. The PR model demonstrated additional CCG cost savings if myCOPD is used for the delivery of PR services, but the results of both models should not be combined due to overlap between patient populations. The PR costing scenario also demonstrated that purchasing a PR service licence for use of myCOPD exclusively for PR delivery is likely to be cost saving, provided sufficient uptake and annual referrals.

4 National Institute for Health and Care Excellence Guidance

4.1 Draft Recommendations

The evidence submitted by the company and the EAG’s critique of the evidence was presented to the MTAC, who considered draft recommendations relating to myCOPD following their meeting in September 2021. These were as follows:

  • myCOPD shows promise for self-managing COPD; however, there is not enough good-quality evidence to support the case for routine adoption.

  • Further comparative research is recommended to address uncertainties about myCOPD’s clinical benefits on outcomes, such as the rate of acute exacerbation and how it affects healthcare resource use such as hospital admissions.

4.2 Consultation Response

During the consultation, NICE received 108 consultation comments from 11 consultees (two company representatives, two HCPs, six members of the public and one professional organisation representative). These are collated into themes and are detailed in Table 3. Further detail can be found on the NICE MTG68 webpage [1]. The comments covered the population groups, clinical evidence, economic inputs and model uncertainty, inequality, the technology, myCOPD engagement and wording clarifications. The draft recommendation did not change after consultation.

Table 3 Consultation comments summary [1]
Full size table

5 Key Challenges and Learning Points

One challenge the EAG and committee came across was understanding the uptake of myCOPD due to the paucity of evidence in the specific populations being modelled. Uptake is of particular importance due to a subscription pricing model. At the time of submission, the cost of myCOPD was applied to the whole CCG population (rather than only individuals with COPD) and therefore the higher the uptake, the more cost saving myCOPD is shown to be. Since the submission, CCGs have ceased to exist, with integrated care systems taking over in July 2022. It is unknown how the pricing scheme currently works and how this will change the licensing system and model results.

A challenge of evaluating patient-facing DHTs is understanding how the level of engagement with an application (such as myCOPD) links to a change in clinical outcomes. Outcomes from the evidence used in the economic models were short term (e.g. three months in the RESCUE study), which makes it difficult to know if the observed benefits would continue with ongoing use of the app or whether they would wane, or alternatively, whether continued engagement is necessary. Yardley et al. emphasised the importance of ‘effective engagement’, which is defined as the engagement necessary to produce the desired outcomes [23]. There are several ways of measuring engagement (rather than the ‘more engagement, the more effective’ approach) and a framework has recently been developed to help link usage to clinical outcomes in DHTs [23,24,25]. These could be used when designing studies for the effectiveness of DHTs, such as myCOPD. Furthermore, clinical experts have suggested that support from the health service may improve patient engagement [1].

Another challenge was the small sample size of the included RCTs. For the two trials evaluating the use of myCOPD for self-management (RESCUE and EARLY), no positive significant benefits were shown for any of the clinical outcomes: CAT score, acute exacerbations or hospital admissions, and there was no significant difference in inhaler errors or HRQoL; however, the sample size in these studies was small (n = 41 and 60, respectively). No power calculation was reported for the RESCUE trials, but the authors noted that the study had limited power to demonstrate effects on all measured outcomes. For the EARLY trial, a power calculation indicated that the trial was suitably powered; however, the authors noted that the small sample size and baseline differences between groups limited their ability to demonstrate statistically significant differences. From these trials, it was therefore unclear whether myCOPD had benefits for the self-management of COPD.

For the trial evaluating the use of myCOPD in PR compared with face-to-face PR (TROOPER), it was reported that the trial was adequately powered and the study showed that myCOPD PR was equivalent to face-to-face PR. The EAG assessed the trial power calculation and determined that the distance used to determine a minimal clinically important difference in the six-minute walking test (40.5 m) was higher than the distance in some recommendations (30 m) [26], resulting in an underestimation of the sample size needed to prove non-inferiority. In addition, the sample size had been calculated based on 58% power, lower than commonly used rates of 80–90%. It was therefore assessed that despite claims for suitable power, the study was likely to be too small to conclusively determine whether myCOPD in PR was equivalent to face-to-face PR.

6 Conclusions

The MTEP evaluation process was followed for the development of medical technologies guidance on myCOPD. This included a submission of clinical and economic evidence by the company, critical appraisal of this evidence by the EAG, drafting of recommendations by the MTAC, and a subsequent consultation. Following this process, the MTAC judged that myCOPD has the potential to help self-manage COPD in adults, but uncertainties in the evidence first need to be addressed [1].