“What our health-care systems need to do with their limited budgets is maximize value and no dimensions of health … will be a good measure of value because people consider other aspects in their evaluations … We must back out of the dead end that is pursuing overall efficiency” [1].

1 Introduction

We are grateful for the opportunity to comment on the results of McDonald et al. [2]. In this note, we summarize their findings, build on their interpretations, and suggest future directions for researchers interested in improving a health technology assessment (HTA) process.

2 Summary of the Findings

McDonald et al. searched for reasons why Canada and other countries might separate their cancer drug HTA process. They appraised whether any of the rationales they found were derived from the assumption commonly made by health economists that “the goal of society or decision makers is to maximize the total aggregate health benefit conferred to a population for a given level of resources”. They found that this rationale was not used by Canada or any country to support the use of a separate HTA process for cancer drugs. The researchers were also not able to find another country besides Canada that had a separate cancer drug HTA process.

These findings may seem surprising: Canada is the only country in the world that has a separate cancer drug HTA process; and McDonald et al. could find no justification based in economic theory for a separate process. To be clear, the authors reported that other countries (e.g. Denmark, Belgium, the UK) set aside a separate budget for cancer drugs, and the authors found several “non-economic” rationales for Canada’s separate HTA process. There are two potential interpretations of these search results: (1) the search missed something or (2) there really is no economic rationale in print for a separate cancer drug HTA process. Although their study might not be considered a methodologically robust systematic review, one could reasonably assume that McDonald et al. found all of the key documents. If one feels comfortable with the study’s findings that no evidence exists, then it is important to consider what the findings really mean.

3 Additional Interpretations

One way to view the findings of McDonald et al. is to consider the context in which the separate cancer drug HTA process was developed. Perhaps Canadian decision makers did not need an economic rationale for them to decide to create their own separate cancer drug HTA system. While it does not appear common for countries to set up separate HTA systems, in Canada, this happens. For example, nearly a decade ago, the province of Ontario created the Committee to Evaluate Drugs-Cancer Care Ontario (CCO) Subcommittee to review cancer drugs and to produce drug funding recommendations for the Committee to Evaluate Drugs (CED). The CED reviewed all drugs and produced funding recommendations for the Ontario Public Drug Programs in the Ministry of Health. For all drugs, both cancer and non-cancer, the funding decision was then made by Ontario’s Executive Officer, taking into account the CED’s recommendations. Thus, Ontario had a separate cancer drug HTA system that Canada used as its blueprint.

Canada’s interim Joint Oncology Drug Review was based on Ontario’s existing review process for cancer drugs, and the interim Canadian process was in place until a permanent process was approved based on an evaluation of iJODR [3]. All participating provinces had access to recommendations made by both the CED and the CED-CCO Subcommittee with respect to the listing of oncology drugs [3]. After more than 3 years of experience from the interim process, the provinces through the Steering Committee of the pan-Canadian Oncology Drug Review (pCODR) created for themselves a permanent, separate cancer drug HTA process, modeled after Ontario’s separate cancer drug HTA process. The resulting process is similar to what Ontario had: the cancer drug funding recommendation from a review committee with cancer expertise (now from the pCODR’s Expert Review Committee and not the CED-CCO Subcommittee) is sent to the CED to make its recommendation before Ontario’s Executive Officer makes his/her funding decision. Thus, Ontario receives separate cancer drug funding recommendations from the pCODR (see Fig. 1), but the provincial funding recommendation is still made by the CED (a committee that reviews all drugs) and the provincial funding decision is still made by the Executive Officer (a person who makes funding decisions about all drugs).

Fig. 1
figure 1

Example of activity (recommendation vs. funding) by level (e.g. pan-Canadian vs. provincial). CDEC Canadian Drug Expert Committee (a pan-Canadian non-cancer drug funding recommendation committee), CED Committee to Evaluate Drugs (an Ontario all drug funding recommendation committee), pERC pan-Canadian Oncology Drug Expert Review Committee (a pan-Canadian cancer drug funding recommendation committee), MOH Ministry of Health (in Ontario all drug funding decisions are made by a decision maker called the Executive Officer)

Full size image

In Ontario, a separate cancer drug HTA system is about who makes the first funding recommendation. Ontario has a predisposition for separate HTA processes, as they also have a separate process for devices and medical technologies in addition to the separate HTA process for cancer drugs. It is possible that the reason the provinces designed and built for themselves a separate HTA process at the national level may be related to the reason that Ontario designed and built for itself a separate HTA process. No economic rationale may have been provided in print or otherwise. However, creating and running a separate HTA process takes a lot of work. The separate process may not have been justified based on a publicly reported economic rationale, but that does not mean it is not justifiable. Is it a paradox that economic rationale is not evident in a process that requires a cost-effectiveness analysis as “economic evidence?” Perhaps there are other objectives that are being optimized, and perhaps researchers should explore this.

This interpretation may stimulate health economists’ desire to be more involved in HTA policy matters. Two options include (1) explaining to decision makers (e.g. through scientific publications) that they are not behaving as economics dictates they should or (2) studying how we can be of assistance and then attempting to do that. McDonald et al.’s article can be viewed as demonstrating the futility of the first option. If the system is not set up with a stated economic rationale, it is likely that policy advice based on economic rationale (that contradicts established policy directions) will fail. Statements doubting that decision makers can be assumed to know their willingness to pay (WTP) because they cannot find answers to the theoretical problems we assume they have (e.g. knowing the incremental costs and benefits of all programs, coping with indivisibilities and non-constant returns to scale in programs) is a convenient way for analysts to fit real-world policy making into their academic paradigm. These statements suggest either ignorance of practical issues facing those who actually do HTAs or willingness to ignore the opportunity to be useful. Technical concerns are correct in theory but may be useless in practice when people have to make recommendations or decisions without this knowledge. This is not a hypothetical situation; it happens every month when recommendation committees or decision makers convene. This leaves option (2) studying how we can be of assistance as a promising direction for future research.

4 Future Research

As McDonald et al. cannot find justification for a separate cancer drug HTA process using one particular economic hypothesis (e.g. that review bodies exist to help decision makers make resource allocation decisions to maximize health), perhaps a new hypothesis is needed. A good alternative hypothesis would explain what we actually do see. New hypotheses could be guided by insights from structured conversations with those involved in the separate HTA process [4, 5] and by building on previous work in the area of the role of economic evidence in Canadian oncology reimbursement decision making [6]. Because the pCODR was designed as a separate process by the provinces for the provinces, it provides clues through revealed preferences on how the provinces wanted to set up the structure and process.

For example, the pCODR requires its Expert Review Committee to create for the provinces non-binding recommendations using a deliberative framework. The framework considers (1) clinical evidence; (2) economic evidence; (3) patient perspectives; and (4) system feasibility. There are no thresholds for any of these dimensions. Moreover, this framework indicates there is more than economic evidence considered for recommendations, suggesting there is more than economic considerations that matter for decision makers too. After the pCODR issues a recommendation, each province can make its own funding decision. In Ontario, the pERC recommendation goes to the CED (which considers all drugs) for a recommendation. Ontario’s Executive Officer has both pCODR’s and the CED’s recommendations when he/she negotiates with the drug company over whether and how the drug will be covered. These negotiations happen behind closed doors and are confidential. As a result, people who know what happens, cannot tell you (by law), and people who tell you what happens cannot know (by law). Additionally, because decisions are made behind closed doors and recommendations are not, the advice one hears about what is useful is from an academic or recommendation point of view, but it is usually not from a decision-making point of view. For example, if there is a threshold that is used for WTP, it may never be known because the real price payers receive is not reported. Without the real price (that changed the cost-effectiveness estimate from > to <WTP), there is no way to know the real WTP. Last, decisions made behind closed doors do not involve the capacity that was there to review the clinical and economic evidence; the audiences are different. Fancy tools may not be useful and subtle distinctions may not be appreciated.

Given this context, why do Canadian decision makers want a separate cancer drug HTA process? If the economics rationale currently being endorsed cannot answer this question, this suggests we need new thinking. The work of McDonald et al. reminds us that economics is about scarcity, choices, and opportunity cost. These concepts apply as well for how we spend our time as researchers in the field of HTA. An economic rationale may be difficult to find for why Canada has a separate cancer drug HTA process, but other valid rationales do exist. Future research could develop a better understanding of what are the structural, procedural, and outcome-based objectives that decision makers desire. Perhaps 100 % transparency is not their main goal. Nevertheless, through observation of their behavior and discussion, it is possible to perceive a revealed preference for a separate cancer drug HTA process. If provinces are assumed to maximize some objective function, then there must be some good that comes from a separate cancer drug HTA process. Beyond the development for HTA of a new culture and innovative methods and processes, the provinces value the pCODR; when the pCODR was recently moved under the governance of the Canadian Agency for Drugs and Technologies in Health, a commitment was made to keep it a separate process for cancer drugs. Cancer drugs are reviewed by the pCODR’s Expert Review Committee and non-cancer drugs are reviewed by the Common Drug Review’s Canadian Drug Expert Committee (see Fig. 1). Future research could study what provinces’ rationales and objectives are for setting up the system they have set up for themselves. Then, we must use our skills as researchers to help them achieve what society has endorsed them to achieve through their role as socially legitimate decision makers.

As HTA researchers, we must continue to develop and apply new methods of analyzing data and displaying information. We must also face the reality that the purpose of our role may be to promote goals related to process rather than outcome, suggesting that getting the question of interest right may be more important for researchers than correctly solving the wrong problem. There is tension between that which is theoretically ideal and that which is practically useful. McDonald et al.’s findings can be viewed as a reminder that our future research priorities should be to explore how to bridge this gap by connecting academically based health economists with problems from reality and practically based decision makers with solutions from theory. In conclusion, McDonald et al.’s findings point to the need for researchers to consider a broader view of what decision makers want and what we can do to make our research more useful. Politicians, not economists, are making strategic decisions for the healthcare system; the rationale driving policy on cancer drug HTA may not be from economic theory (or it might but decision makers may have a more complex objective function). We should endeavor to optimize given these binding constraints.