Background

Economic evaluation is the comparative analysis of alternative technologies with respect to their costs and consequences [1]. It is often used late in the evaluation of drugs, devices, and other technologies to inform coverage decisions [2]. This is problematic because the evidence portfolio at the time of the reimbursement decision is often incomplete. Cost and preference data are often not gathered prospectively within clinical trials designed to evaluate effectiveness. This means that analysts cannot directly use trial data and must struggle to assemble evidence from a variety of sources and settings to evaluate the cost-effectiveness of new interventions [3]. Published data suggest that considerable economic uncertainty existed in about half of the submissions for reimbursement decisions [4]. Also, clinical trial design is rarely informed by ex ante economic modeling. This means that optimal strategies and the full range of subgroups may not have been fully considered prior to commencing the clinical research program.

Economic evaluation can be used early in the process of technology evaluation. As an example, Vallejo-Torres et al. illustrate a three-stage cost-effectiveness analysis of a new medical device: an early phase in which simple methods are used to estimate the maximum price attainable for the technology, a mid-stage which synthesizes data into a cost-effective model and identifies which information is most valuable to reduce decision uncertainty, and a late stage in which all relevant information is synthesized [5]. The results suggest that early and iterative economic evaluation could be useful to inform decisions along the technology development process.

Early evaluation can contribute to decision-making by both industry and government [6, 7]. From an industry's perspective, early evaluation may be used for early market assessment, managing research and development portfolios, and informing pricing and reimbursement scenarios. From the policy perspective, decision makers may benefit from information supplied by early evaluation. Thus, there is interest from both innovators and payers in early health technology assessment and, more specifically, early economic evaluation, to inform planning and development decisions by industry and to inform the potential of new technologies that may meet health system needs [8].

Though interest in early economic evaluation is considerable, much applied research is still at the pilot stage. There is uncertainty about which technologies can benefit from early evaluation, what methods are appropriate, and the contribution of early economic evaluation to health technology assessment [8]. The aim of this systematic review is to describe the characteristics of early economic evaluation of emerging technologies and to understand current methods to early evaluation.

Methods/design

This protocol describes a systematic review of economic evaluation studies that are conducted to inform the development and planning of early evaluation of emerging health technologies.

Inclusion criteria

An economic evaluation is defined as the comparative analysis of alternatives with respect to their associated costs and health consequences. A study is included if it reports an early economic evaluation of a regulated health technology in order to inform the development of the technology (e.g., pre-clinical research or phase I or II studies). For this review, regulated health technologies include pharmaceuticals, biologics, medical devices, or biomarkers. The term ‘biologic’ refers to a class of therapeutics that are produced by means of biological processes, including, for example, the blood production-stimulating protein erythropoietin, a growth-stimulating hormone, a gene therapy, a monoclonal antibody, or a vaccine [9]. The term ‘medical device’ refers to a class II device (e.g., blood pressure monitors, contact lenses, pregnancy test kits, single-use surgical instruments, catheters), a class III device (e.g., ventilators, cardiac monitors, hip implants, knee implants, lasers, chlamydia test kits, glucose meters), or a class IV device (e.g., defibrillators, pacemakers, coronary stents, HIV test kits, neurosurgical shunts) that requires product licensing for general marketing purposes (examples are from a presentation by Ms. Sarah Chandler of Health Canada to MaRS-Excite, September 2012, with permission). The term ‘biomarker’ (or a biological marker) refers to an objectively measured and evaluated indicator of a normal biological process, a pathogenic process (e.g., a DNA sequence that is associated with susceptibility to disease), or a pharmacologic response to a therapeutic intervention [10].

Literature search

The literature search will be conducted on multiple databases, including MEDLINE (Ovid), MEDLINE In-Process (Ovid), EMBASE (Ovid), the Centre for Reviews and Dissemination: National Health Services Economic Evaluation Database (NHS EED), the Cochrane Library: Health Technology Assessment Database (HTA Database), and EconLit. The first four sources are standard repositories of economic evaluation studies. The last database is a reliable source of citations and abstracts of economic research [11]. All databases will be searched from the date of inception until present.

The search used both Medical Subject Headings (MeSH) as well as keyword variants of all relevant terms, including keyword terms derived from MeSH scope notes to increase retrieval of citations from Medline In-Process. The search developed in MEDLINE will be adopted with the syntax and subject headings appropriate for the other databases mentioned. In addition to early economic evaluation keywords and MeSH terms generated through scoping search (below), the final search strategy will include the validated NHS EED search filter and a published MEDLINE G and EMBASE G filter to identify potentially relevant citations of economic evaluation studies in MEDLINE and EMBASE [12]. Both MEDLINE G and EMBASE G filters were tested and showed that they improve the search precision [12]; however, both have been modified to conform to this study's needs. Namely, the subject heading for randomized control studies (RCTs) was removed from original filters, since we did not want to limit the retrieved studies only to RCTs. Limits that are part of NHS EED, MEDLINE G, and EMBASE G filters included humans and publication type. The final search was not limited by language.

Since the early economic evaluation concept is not well defined and it is not included in the MeSH controlled vocabulary, an initial scoping search was performed in order to identify potential candidate keyword terms as well as MeSH subject headings. A set of seed articles was used to develop the MeSH terms and keywords used in the literature search [2, 5–8, 13–22]. These articles were selected because they are commonly cited and highly relevant to early health technology assessment or early economic evaluation. The relevant MeSH terms harvested from the seed articles include ‘device approval,’ ‘diffusion of innovation,’ and ‘decision support techniques.’

From the seed articles, we extracted the cited keywords and other keywords that are deemed relevant to the aspects of early economic evaluation. Table 1 describes these aspects and the associated keywords. The first aspect is that the evaluation is conducted at an early phase in the development of the emerging technology. The objective of an early evaluation is to examine the likely cost-effectiveness of the technology. The evaluation is to provide decision support for product and policy development. There is limited supporting evidence regarding the effectiveness of the technology. Lastly, early evaluation may have an influence on the design and future performance of the technology.

Table 1 Aspects of early economic evaluation and associated keywords used in the literature search
Full size table

The search strategy has been developed by an information analyst with the review team and will be peer-reviewed by another information analyst from a systematic review research group [23]. Peer-reviewing of the literature search strategy will be conducted according to the Peer Review of Electronic Search Strategies (PRESS) statement [24]. The protocol for this systematic review is registered at PROSPERO, the International prospective register of systematic reviews (registration number CRD42013004636) [25].

Additional citations will be identified via scanning the reference lists, author searching of potentially relevant studies, and forward citation searching in Scopus. We suspect that many early economic evaluation studies are unpublished, especially those conducted by industry for internal use only. There appears to be no straightforward way to identify authors of these unpublished studies. We will therefore consider a chain-referral sampling approach to sample ‘hidden populations’ in our attempt to contact these authors [26].

In the chain-referral sampling approach, a hidden population is one in which a sampling frame (i.e., a list of all the members of the population) cannot be constructed in advance. As an alternative, new members are selected from the social network of existing members of the sample. In this approach, first a number of seeds are selected [26]. These seeds are members of the hidden population that have been identified: authors of published early economic evaluation studies who work for industry, and similar individuals identified from our contact files. The seeds are contacted and form the first stage (stage 0) of the sampling process. The seeds identify other members of the population. The members identified in the next stage (stage 1) are approached and then asked to identify other members. This process is continued until no new members could be identified.

We fully recognize that the above sampling scheme may not yield a comprehensive collection of unpublished study reports. The key barrier to access to internal reports of early evaluation may be the reluctance to share proprietary information. Possible authors will be contacted by email. We will follow up by phone with authors who contributed to unpublished studies in order to discuss their involvement and, if possible, request information about the unpublished material. For this approach, non-responders are defined as a lack of response after three attempts to contact a member of the sampling population.

Study selection

Due to the broad inclusion criteria, screening of potentially relevant citations will be conducted by pairs of reviewers who are formally trained and experienced in economic evaluation. Full-text reports of citations that are deemed relevant by one screener will be obtained. Reasons for exclusion will be captured for studies that are deemed to be not relevant for the review. Full-text review will be conducted in pairs; disagreement will be discussed and, if necessary, adjudicated by another health economist.

Data abstraction

Table 2 describes data elements that are going to be extracted from the included studies. First, understanding the decision context is necessary to discern the contribution of early evaluation to decision-making in product and policy development. Data pertaining to the decision context will be abstracted (e.g., who initiates the evaluation, who commissions it, the funding sources, why an evaluation is needed right now, who conducts the evaluation, and the primary target audience). Other abstracted data include the decision problem, disease classification, and target population, technology type (e.g., pharmaceuticals, medical devices, biomarkers), and technology stage (e.g., pre-clinical, phase I or II studies). Data pertaining to the methodological aspects of an early evaluation will be abstracted, especially how authors of included studies handled uncertainty in the early evaluation (e.g., probabilistic sensitivity analysis, value of information analysis).

Table 2 Abstracted data from included studies
Full size table

Open text coding will be used to capture the following descriptions: the likely effectiveness of a new technology and how the associated health consequences and resources used were measured and valued. These descriptions will be used to discern the data sources, methods, and assumptions involved in the early evaluation. We will assess as to whether the results of an included study are adequate to answer the main study questions or objective, according to authors of included studies and according to the reviewers. A pair of reviewers using the Dedoose software for qualitative literature review will abstract each included study independently [31].

Evidence synthesis

Characteristics of early economic evaluation studies will be summarized according to the various aspects that define early evaluation (Table 2). Given the qualitative nature of this systematic review, there will be no quantitative evidence synthesis. Unpublished studies obtained from the chain-referral sampling will be summarized separately because the representativeness of this sample of unpublished studies (relative to the population of early economic evaluation conducted to support internal decisions) is uncertain.

Discussion

This protocol describes a systematic review of published and unpublished studies of early economic evaluation of regulated health technologies, when the supporting evidence for their effectiveness is not fully established. The aim is to understand current practice regarding the use of early economic evaluation to inform decisions in the development of emerging technologies, including decisions regarding the allocation of private and public resources for the development of these technologies and decisions regarding the planning and design of research programs to collect additional evidence to support regulatory and reimbursement requirements. This systematic review will summarize the methods used in early economic evaluation and provide data regarding the contribution of early economic evaluation to decision-making for product and policy development.

Our systematic review is timely. Conceptual discussion of early HTA over a decade ago has evolved into concrete implementations over recent years [6, 21, 32–34]. As we understand it, early HTA initiatives now exist in the UK, the Netherlands, Austria, and Ontario [35, 36, 21, 37]. These initiatives facilitate the collaboration between private enterprises, public sectors, and academia in the development of selective technologies that address the needs of patients and health systems and, to varied extent, investment expectations [35]. Early economic evidence may facilitate decisions regarding the selection of promising technologies and the complex trade-offs between public and private investments [36].

Economic evaluation is essential for health technology assessment. Ijzerman and colleagues review previous work and methods of early health technology assessment and propose a conceptual framework for early assessment that acknowledges the different phases in technology development and the specific decision problems faced by different decision makers [8]. They review a range of tools to identify and evaluate technologies in development, most notably early health economic modeling. They suggest that early HTA is an emerging field of research that will likely become more important because of the increasingly complex trade-offs between investments and access in medical product development.

The method outlined in this protocol is sometimes referred to as a meta-epidemiological investigation. In particular, there are well-established precedence regarding the use of systematic review to investigate factors affecting the conduct and reporting of trials, systematic reviews, and economic evaluations of health interventions [38–45]. By adapting a systematic approach to our investigation, our aim is to collect relevant data from a representative sample of all early economic evaluation studies and, therefore, reliably discern the contribution of early economic evaluation to product and policy development.