FormalPara Key Points for Decision Makers

Health technology assessments (HTAs) are typically performed as one-off evaluations and can quickly become out-of-date.

Living HTA approaches can ensure that the HTAs are up-to-date, and potentially living HTAs could be updated manually or (semi-)automatically using innovative software platforms.

However, living HTA involves substantial time, planning and resource commitments, and as such should only be used in situations where it is important to ensure the HTA is up-to-date.

1 Introduction

Health technology assessment (HTA) agencies perform evaluation of clinical and cost-effectiveness evidence of new interventions to decide whether they should be reimbursed. These are typically performed as one-off evaluations and can potentially become out-of-date due to various reasons (e.g., availability of new data, new comparators, new methods, etc.). While some HTA agencies perform updates of HTAs periodically if certain criteria are met, these updates are typically a few years apart and the results of updates may already be out-of-date by the time of the publication.

There is growing recognition of the need for the HTA process to respond to an evolving evidence base, particularly in reimbursement decisions with high uncertainty. A recent paper on ‘Life-cycle HTA’ suggests that HTA must explore the value of health technologies from inception through maturity, and proposes a model for integrating changes arising from new evidence to feed into adoption, no adoption and disinvestment decisions [1]. However, as far as we know, there is no literature on the practicalities of performing a responsive, dynamic HTA (which we call ‘Living HTA’).

While examples of living systematic reviews and living meta-analyses are already well established, living HTA is not yet fully defined or understood. In this paper, we outline the ways in which the different parts of HTA can become? out-of-date, and provide a definition for living HTA and the situations in which it could be useful. While there are similarities between living HTA and the updates that HTA groups make, we outline how the living HTA process could potentially be operationalised from the outset, provide recommendations regarding when its implementation may be necessary, and describe potential triggers for an update. We also outline the practical implementation of living HTA, using a manual approach to operationalise living HTA or leveraging recent innovations to operationalise living HTA using semi-automation. Finally, issues and challenges with planning and implementing living HTA are considered.

2 Static Health Technology Assessments (HTAs)

Traditional HTA performed as one-off evaluations use a one-time snapshot of the evidence base (usually the last date of the literature review searches), which can produce recommendations that may not reflect the most current evidence. An illustration of how the current (static) HTA process can quickly become out-of-date is presented in Fig. 1 and is described in more detail below.

Fig. 1
figure 1

Critical steps involved in HTA and examples of how static HTA can become out-of-date*. HTA health technology assessment. *Note that this figure presents some common reasons for the HTA process to be 'out-of-date' but it is not an exhaustive list. In addition, latter stages in the HTA process can become out-of-date for all the reasons in the earlier stages (as the reasons for being out-of-date are cumulative, i.e. they 'build up' from left to right)

Full size image

2.1 Clinical Effectiveness (Evidence Synthesis and Meta-analysis)

2.1.1 Current Process for Evidence Synthesis and Meta-analysis

Systematic reviews and meta-analyses are essential tools for synthesising evidence to inform the clinical decision making that HTA underpins. Systematic reviews summarise available evidence based on a thorough review of the literature followed by rigorous critical appraisal to synthesise the findings on a specific topic [2,3,4]. Meta-analysis uses statistical techniques for combining quantitative data from several independent studies identified in the systematic review to produce a single clinical-effectiveness estimate [2].

2.1.2 How Clinical Effectiveness Evidence Can Become Out-of-Date

Systematic reviews frequently take between 1 and 3 years to complete [5] and the searches are often out-of-date at the time of publication [6,7,8]. Conclusions from clinical effectiveness assessments can become out-of-date if there is new evidence since the publication of systematic reviews and/or new comparators. New relevant evidence could be in the form of availability of data with longer follow-up for the previously included studies or publication of new studies after the literature searches, which occurs at the onset of the systematic review process. If the systematic review is out-of-date, then the pooled result from the corresponding meta-analysis would also be out-of-date.

2.2 Cost Effectiveness (Health Economic Modelling)

2.2.1 Current Process for Cost-Effectiveness Modelling

Health economic models are used to compare the costs and consequences of alternative options to estimate the cost effectiveness of a health technology of interest [9]. The structure of these models is based on the clinical condition, and the models use synthesised evidence on natural history/disease progression, clinical effectiveness, costs, and utilities as inputs to inform cost effectiveness for reimbursement decisions [9]. It should be noted that long-term data and real-world evidence often do not exist when the health economic model for HTA is developed.

2.2.2 How Cost-Effectiveness Evidence Can Become Out-of-Date

The model structural assumptions and the model inputs can become out-of-date over time, as well as the methods used to estimate the cost effectiveness [10]. Examples of changes related to the model structure include a new technology becoming standard of care, or additional knowledge around the clinical condition that would influence either the model health states or structural assumptions. Model input changes include longer follow-up trial evidence on treatment effectiveness and safety events, or real-world evidence on discontinuation, safety, or other outcomes. Additionally, methods for cost-effectiveness modelling can change over time, such as changes in the threshold, discount rate, or perspective.

2.3 HTA Report

2.3.1 Current Process for Developing HTA Reports

An HTA report is typically released around, or soon after, the time of regulatory approval of an intervention to inform the reimbursement decisions. The report includes clinical effectiveness evidence (from the evidence synthesis) and cost-effectiveness evidence (from the health economic modelling) along with a summary of key results and recommendations.

2.3.2 How HTA Reports Can Become Out-of-Date

As discussed above, once either the clinical effectiveness evidence or health economic modelling are out-of-date, the conclusions of the entire HTA can be compromised. The report can be out-of-date if the set of potentially relevant interventions and comparators changes since the time of publication. For example, new interventions can enter the marketplace, and the comparator can be replaced by a new standard of care. Similarly, the report can be out-of-date if there are new or updated data to inform the inputs to the cost-effectiveness model, e.g., if there are long-term data to inform the survival beyond the trial period or if there are new data on the costs or utilities.

3 Living HTA

Most people understand ‘living’ research to be a process that is updated periodically [11,12,13], but there is no consensus on definition and the methods involved. Furthermore, living systematic reviews, living meta-analyses or living health economic models are sometimes erroneously referred to as living HTAs, despite not covering the full HTA process (which includes literature searching, systematic reviewing, meta-analysis, health economic modelling and HTA reporting).

We define living HTA as a full HTA that is planned from the outset to be updated at regular intervals or at specific trigger points (e.g., in light of new evidence and/or feedback from stakeholders). Table 1 provides a brief overview of the key characteristics of living HTA.

Table 1 Brief overview of key characteristics of living HTA
Full size table

4 Operationalising Living HTA

To our knowledge, there is no published example of living HTA, therefore we present an idealised version of living HTA by combining parts of the HTA process that have been made ‘living’. We outline two approaches for the practical implementation of living HTA: (1) using a manual approach to operationalise living HTA; and (2) leveraging recent innovations to operationalise living HTA using (semi-)automation. In the manual living HTA, each of the HTA parts are made ‘living’ using manual updates, and the different HTA parts are also combined manually. In the (semi-)automated living HTA, each of the HTA parts are automatically updated and they can be combined using automation (i.e., fully automated living HTA) or manually (i.e., semi-automated living HTA).

4.1 Manual Living HTA

To operationalise manual living HTA, we suggest ‘manually’ combining the different parts of the HTA process that have been made ‘living’. We present examples of living systematic reviews and meta-analyses [14], and living health economic models that are updated manually [15]. We then present the need for standard templates to transfer the data between the different steps to ensure the smooth running of the ‘manual living HTA’. Figure 2 presents a brief overview of manual living HTA.

Fig. 2
figure 2

Manual living HTA. HTA health technology assessment

Full size image

4.1.1 Living Systematic Reviews

There are now several examples of ongoing living systematic reviews in which the literature searches are updated periodically and new evidence incorporated accordingly [16, 17]. Updating electronic searches can involve the use of manual searching and/or auto alerts [18, 19]. Performing updates to grey literature searches such as registries and websites is likely to be challenging and time-consuming compared with electronic searches. In some cases, it may be necessary to revise the original search by looking at the yield of potentially relevant records and the appropriateness of the sources searched. Once all the citations are collated, they are screened manually by reviewers to select the relevant studies for data extraction and quality assessment.

4.1.2 Living Meta-analysis

The living meta-analysis presents a pooled result of the most recent available evidence, repeating the same statistical method/model after adding the newly identified studies. Simmonds et al. [20] summarises the methods that could be used to avoid type I error inflation, but cautions some issues relating to the use of these adjustment methods remain uncertain and further research is required.

4.1.3 Living Health Economic Models

There has been substantial conversation about model transparency and open source modelling in the field of health economics [15, 21], but progress has been lacking as to how an economic model can be made ‘living’. A living health economic model is one that is updated over time as new evidence emerges. Early living health economic models have varied from fully open source models to models that are not made publicly accessible but updated solely by the model developer and from which only the final updated outputs are presented.

4.1.4 Operationalising Manual Living HTA

It is important to note that in manual living HTA, all the different steps, along with the combination of the steps, are all performed physically by researchers. To operationalise living HTA, it would be helpful to develop and share standard templates for data extraction, meta-analysis outputs, model inputs and model results. In particular, an economic model that has a standard template (and process) for entering model inputs/settings and to output the results would be useful [22]. The HTA report then needs to be updated with the latest clinical and cost-effectiveness evidence results, as well as an updated executive summary and discussion/conclusions based on the new findings.

4.2 Potential for (Semi-)Automation of Living HTA

Recently, there have been technological innovations to (semi-)automate parts of the HTA process that indicate potential to achieve real-time living HTA by building on and combining these technologies. It should be noted that these tools have not been formally evaluated, or currently accepted, by HTA agencies.

Figure 3 presents an example for potentially operationalising a living HTA using (semi-)automation. There are many tools available for automating elements of HTA, however we present examples of open-source technologies to automate literature searching (e.g., RobotSearch), systematic reviews (e.g., RobotReviewer), meta-analyses, and health economic modelling (e.g., using R software). We then present the need for secure web-based user interfaces (e.g., R-Shiny [23]) to transfer the data between the different steps to ensure the smooth automatic updating to achieve real-time living HTA. It may not be feasible to automate completely as it is important to have manual input from HTA researchers to sense check the updates and to ensure that the outputs of each stage are appropriate.

Fig. 3
figure 3

Potential example of living HTA using (semi-)automation. HTA health technology assessment

Full size image

4.2.1 Automated Literature Searching

Custom tools can be used to mechanise parts of the literature search process such as sifting. For example, RobotSearch [24] is an open source machine-learning tool that identifies randomised controlled articles (RCTs) from PubMed search results. To identify the RCT studies without the requirement of manual screening, RobotSearch uses a high recall and validated machine learning classifier that is capable of ordering retrieved papers by study sample size and quality (i.e., risk of bias). Similar tools offer efficiency in the searching and study selection stages, where algorithms built in programmes such as Python can narrow down potentially relevant articles in fields where many thousands of articles are published (e.g., HIV [25]).

4.2.2 Automated Systematic Reviews

Custom tools can also be used to mechanise parts of the systematic review process to produce semi-automated systematic reviews. Collaborative initiatives have developed open source software to support researchers in evidence synthesis by using machine learning for data extraction (such as RobotReviewer [26]). This is a rapidly developing field that could potentially reduce the time and resource burden that living systematic reviews inevitably require to sustain on a long-term basis.

4.2.3 Automated Processes for Combined Literature Searching and Systematic Review

Trialstreamer [27] combines both trial identification (using RobotSearch [24]) and data extraction (using RobotReviewer [26]) to maintain a new and publicly available living database of all healthcare RCTs in humans, from PubMed and the WHO International Clinical Trials Registry Platform. The tool utilises population, intervention, control and outcomes (PICO) search strategies for identifying and classifying RCTs, as well as extracting and summarising the key information from the RCTs retrieved.

4.2.4 Automated Meta-analysis

Software environments such as R could be used to facilitate automation of the meta-analysis process. Given the updated input data are arranged in a consistent way that allows an R script to load the data and perform the required analysis, the same R script could be employed to analyse the updated data. For example, the COVID-NMA initiative [28] launched in March 2020, is a living systematic review of coronavirus disease 2019 (COVID-19) trials where the review is updated weekly and the meta-analysis is updated every 2 weeks using ‘metaCovid’ [29], which is freely available as an R-shiny application, allowing users to explore the data and conduct the analysis tailored to their needs.

Automating complex evidence syntheses such as network meta-analysis (NMA) and population-adjusted indirect treatment comparisons require more careful consideration than a pairwise meta-analysis. For NMA, the application should allow users to add and remove interventions, perform inconsistency checking, and conduct subgroup analysis or meta-regression to explain heterogeneity. In the case of population-adjusted indirect comparisons, there should also be flexibility for choosing specific baseline covariates to match.

4.2.5 Automated Health Economic Models

Automatic model updates can be scheduled to run at a set time or when triggered by an event using script-based programming languages (such as R or Python) or via web-based interfaces [30], where the model will run using parameter inputs and data located in the paths provided. These web-enabled platforms are separate to the health economic models (e.g., Microsoft Excel files or R code), which could help address potential intellectual property concerns. There are now several examples of health economic models with web-based user interfaces where the users can amend model inputs and view the subsequent impact on the model findings and conclusions in real-time. Examples of these include models hosted online using the R Shiny platform interfaces [30, 31] and the ICER interactive modeller platform hosting dozens of models [32].

Automated updates are most likely to be undertaken for input parameters only as it is relatively simple to automate to reflect changes in input parameter values (e.g., a new cut of trial data). However, it is much harder to automate changes in structural assumptions since this typically requires adaptation of the model source code.

4.2.6 Automated HTA Reporting

Packages such as R Documentation and RMarkdown [33] (under the R framework) could be utilised for automating HTA documentation. These packages allow the results (i.e., text, figures, and tables) from systematic review, meta-analysis and health economic modelling to be automatically updated in the report when the data underlying the analysis are updated. In addition, RMarkdown consolidates code and documents into a single file (negating the need for copy-pasting results from one place to another), allowing for automated, transparent and completely reproducible archives of work. RMarkdown documents can be ‘parsed’ to produce reports in a variety of different formats, including, HTML, PDF, MS Word, Open document type, etc. Recently developed R package ‘bookdown’ further enhances the reporting and allows for the development of long reports/books with features such as cross-referencing to different sections, tables, and figures [34].

4.2.7 Operationalising (Semi-)Automated Living HTA

Automated HTA can loosely be defined as a system by which HTA updates are scheduled to run automatically at a set time or when triggered by an event. Script-based programming languages (such as R or Python) could be used to link the different steps in the HTA process and scheduled as any other program (e.g., an update to your computer). This would be relatively straightforward to set up, but has not yet been implemented on a live HTA project. It is important to note that semi-automation raises issues such as ethical considerations and potential for errors in automated processes [35]. Furthermore, none of the HTA agencies currently accept automated processes (e.g., automated systematic reviews) in HTA submissions.

5 Issues and Challenges

We have outlined two different approaches for living HTA: (1) manual living HTA; and (2) living HTA using (semi-)automation, each of which has specific issues and challenges compared with the traditional one-off static HTAs performed currently. Table 2 presents a brief comparison of the static HTA approach, manual living HTA approach and (semi-)automated living HTA approach. The pragmatic limitations and implications of implementation are discussed in the following sections.

Table 2 Comparison of static HTA, manual living HTA and (semi-)automated living HTA
Full size table

5.1 Standardisation/Interoperability

For living HTA to be successfully realised, it is crucial that the analysts involved have a common understanding of how the various components of HTA connect to one another. For example, modellers should build models that can be run using an agreed meta-analysis output structure, and statisticians should ensure that their meta-analysis outputs remain consistent to this agreed structure. If the output nomenclature of meta-analysis is standardised, the analysts can easily input the data manually into the health economic model (for manual living HTA) or the model can directly read the results from meta-analysis using software (for semi-automated living HTA). Standardisation of inputs and outputs across HTA sub-disciplines, also known as data interoperability, has a huge potential for living HTA.

5.2 Responsibility, Ownership and Publication

Management of an ongoing update process, such as in living HTA, is likely to involve continued ownership of the research topic. It is likely that each living HTA would require a designated webpage or website, however most institutions do not currently have web templates to support the research software to underpin semi-automated living HTAs. If a designated webpage/website for the living HTA is maintained, previous versions of the living HTA should be archived as supplementary appendices. Furthermore, some academic publishers have already embraced the living systematic review model and it is possible that this could be extended to living HTAs.

5.3 Resource/Time Commitments and Skills

The resource commitments for ‘Living HTA’ are likely to be considerably larger than the status quo, especially in instances where updates cannot be semi-automated. In instances where updates can be semi-automated, high initial sunk costs in establishing a code-base for the living HTA framework could be recouped by lower ongoing costs. However, semi-automated living HTA is likely to require considerable upskilling (e.g., to build and review script-based models that would be utilised in a semi-automated framework).

Whether decision makers and other stakeholders are willing to pay the additional cost to receive living HTA is uncertain. Most academic researchers are funded by fixed-term contracts and therefore living HTA cannot exceed the period for which designated researchers are salaried to work on it. However, living HTA does not have to mean that these research projects live ‘forever’; they can be live for a finite number of years during a period of uncertainty in the evidence or until such a time that a stable conclusion to the decision problem can be reached.

The individuals responsible for the update may not have to be HTA agencies in all instances. Academic groups or consultancies could lead these efforts, or share the responsibility and effort with HTA groups. Additionally, national HTA agencies could work collaboratively to share resources and minimise duplication of similar updates being undertaken in different settings.

5.4 Confidential Pricing Data

HTAs often involve commercially sensitive data such as patient access schemes. Traditionally, these are redacted in static documents that are made publicly available, but the possibility of more frequent updates in a living HTA approach could lead to more frequent changes (e.g., commercial discounts for technologies and their comparators), cascading to make each reassessment more complex and less transparent than the last. A potential solution to this issue is that HTA agencies require consistent, ‘clean’ results to be provided during each reassessment, using all technologies’ public list prices as at the time of the first assessment. Such analyses would not be suitable for decision making as the prices would not reflect the true cost to the healthcare system at the time of reassessment; however, they would help to make the magnitude of effect as transparent as possible, and not lost in the noise caused by any simultaneous price changes.

5.5 Data Security, Copyright and Intellectual Property

The copyright or intellectual property of the work is more complex in a living HTA. Separate from cost inputs, some clinical, epidemiological, and quality-of-life inputs may be provided as confidential at the time of the initial assessment, which can limit the transparency potentially needed to conduct the reassessment and the ability for automation. In addition, there may be issues with data management policies, especially for automated living HTA (which requires storing data in the cloud). Thus, consideration of data security, copyright and intellectual property issues need to be discussed at the outset by all stakeholders to ensure that they are aligned with the institutional requirements.

5.6 Stakeholder Involvement

Agencies need to consider the extent of stakeholder involvement in their living HTA updates. Decision-making panels, such as NICE’s independent Appraisal Committees, need to be involved in the update of their previous decisions in light of new evidence, and therefore require some level of ‘sign off’ for any new recommendations. Clinicians and patients may be able to provide important insights into current practice since the previous HTA. Manufacturers of the technology of interest and its comparators will also likely seek to engage in any living HTA reassessment. Depending on the scale of update (e.g., using the same model structure that was accepted previously and amending only a small number of inputs that have changed in a meaningful way since the last assessment, or a ‘full’ HTA update with entirely new clinical and economic evidence), the level and type of stakeholder involvement can be decided. Living HTA will require conversations with stakeholders to help determine the need for an update, the intervals of the update, the triggers for an update, the type of update and the level of stakeholder engagement that may be needed with each update.

5.7 Impact on Policy/Decision Making

In the current static HTA approach, the agencies provide evidence-based recommendations about health technologies near the time of regulatory approval. Recently, there has been a push towards a ‘life-cycle’ approach to HTA, where the recommendations are updated regularly in light of new evidence [1, 36,37,38,39]. Living HTA can help decision makers in this regard with a frequently updated, living approach to HTA. However, HTA recommendations that change frequently may be difficult for the healthcare system as, at a practical level, it takes time to procure and supply medical technologies that are newly considered to be cost effective, and to use up existing supplies of technologies that are no longer deemed cost effective. It may also be confusing for clinicians and patients if recommended treatments change often, and HTA processes are less impactful if payers cannot act upon them and other stakeholders lose trust in them. Furthermore, a living HTA approach necessitates a commitment to disinvest in technologies that are no longer cost effective and remove them from clinical practice. To understand the potential policy impact of living HTA, readers are referred to a paper outlining ‘Life cycle HTA’, which suggests a framework for constant HTA process to feed into reassessment, adoption, and no adoption decisions [1].

6 Conclusions and Considerations for Implementation

Living HTA can be defined broadly as one that is planned from the outset to be updated at regular intervals or at specific trigger points based on feedback from the stakeholders. The methods for practical implementation of living HTA and the responsibility for ownership and updating require careful consideration, as living HTA approaches are likely to involve substantial time and resource commitments.

Those interested in piloting a living HTA, such as HTA agencies and healthcare decision makers, should ensure that the decision problem requires a living approach to justify the time and resources required. They should consider what update frequency is required and feasible, or whether the updates should be performed at specific trigger points, and ensure that the stakeholder meetings and reimbursement decision-making timelines align with the living HTA updates. The decision between a manual or semi-automated approach to living HTA should be made by the research team in consultation with stakeholders, and ongoing researcher time needs to be factored across the lifespan to ensure that the HTA can be updated appropriately at the chosen time points. Input from skilled HTA analysts is needed to ensure the rigour of updates, and, in addition, semi-automated approaches require advanced software expertise. Ownership and copyright of the final outputs should be considered in advance, and data management infrastructure (e.g., software or cloud-based services) may be necessary if the living HTA is hosted on a dedicated website.

These additional considerations mean that extra funding will almost certainly be required and thus living HTA should be considered at the time of research commissioning. Living HTA should only be used when there is clear need (e.g., in high-priority therapeutic areas), and HTA decision makers should consider close collaboration and shared learning on living HTA, to minimise duplication of efforts.