“Would you tell me, please, which way I ought to go from here?”

“That depends a good deal on where you want to get to,” said the Cat.

“I don’t much care where—” said Alice.

“Then it doesn’t matter which way you go,” said the Cat.

(Lewis Carroll, Alice in Wonderland)

In their critique of our paper, Hoch et al. [1] ask if it matters whether Canada’s separate health technology assessment (HTA) process for cancer drugs has an economic rationale. They say that the answer is no. However, as we will show later, on the basis of their actual comments, there seems to be ambiguity in what this ‘no’ really means. Hoch et al. also suggest that there were other rationales and objectives set for the establishment of a separate HTA review process for cancer drugs. Unfortunately, they do not identify or explore what these other rationales and objectives might have been, nor do they discuss what these are now and moving forward. Hence, we are left in the dark as to what these might be. Their response reminds us of the famous quotation (above) from Alice in Wonderland. We are left not really knowing where the authors want to go and why.

Since we do not know what alternative rationale and objective(s) the authors have in mind, we focus the remainder of our response on comments that we think relate to the subject of our publication (i.e. whether an economic rationale has been provided for the establishment of a separate reimbursement review process for cancer drugs). Specifically, we address the following three issues. First, we discuss whether Hoch et al. really think that an economic rationale is not “needed to justify the creation and adoption of a separate cancer drug HTA process” and what appears to be confusion in their comments between an economic rationale (referred to in our paper as “a rationale that is consistent with an economic perspective” [2]) and goal(s) (or objectives) to achieve through allocation of resources (e.g. to improve population health). Second, we address Hoch et al.’s suggestion that “decision makers may be pursuing different objectives from academic researchers.” Third, we discuss Hoch et al.’s prescription for how to make “positive contributions that help decision makers address the challenges that decision makers have.”

1 The Question of Whether an Economic Rationale is Needed and the Confusion Between an Economic Rationale and a Goal to Achieve via Allocation of Resources

Hoch et al. raise the question of whether an economic rationale is required to support a separate cancer drug HTA review process. However, it seems that they confuse an economic rationale (i.e. maximization of stated goal(s) from available resources), which they seem to support, with the goal that is typically stated as the underlying premise of cost-effectiveness analysis (i.e. maximization of the total aggregate health benefit conferred to a population for a given level of resources), to which they seem to object. In our paper, we described an economic perspective and explained why it is an appropriate perspective to use when asking whether there is a rationale for the establishment of a separate drug reimbursement review process for cancer drugs. This perspective is based on three fundamental principles: scarcity (whatever resources are available, they are insufficient to support all possible activities), choice (because resources are scarce, we must choose between different ways of using them) and opportunity cost (by choosing to use resources in one way, we forego the opportunity to use these same resources in any other way). As noted above, Hoch et al. do not appear to object to these principles, which are embedded in an economic perspective. For example, their opening quotation defines the challenge facing healthcare systems in economic terms (i.e. noting that healthcare systems need to maximize value with their limited budgets). Hoch et al. also invoke an economic perspective in providing advice on how researchers in the field of HTA should allocate their time (i.e. “The work of McDonald et al. reminds us that economics is about scarcity, choices, and opportunity cost” and “These concepts apply as well for how we spend our time as researchers in the field of HTA”). Therefore, Hoch et al. appear to recognize the relevance of applying an economic perspective to guide decision making in situations of resource scarcity.

While they are not clear, what Hoch et al. appear to be challenging is the frequently cited healthcare system goal of maximizing the total aggregate health benefit conferred to a population for a given level of resources. This is a goal that economists typically adopt when healthcare decision makers do not explicitly state the goal they are trying to achieve in allocating scarce resources. As described in our paper, we adopted this goal on the basis of the fact that the Pan-Canadian Oncology Drug Review (pCODR) uses cost-effectiveness analyses (CEA) to make reimbursement recommendations. While Hoch et al. may disagree with the assumption that ‘health’ is what healthcare systems aim to maximize, the applicability of an economic approach does not depend on the goal chosen. Therefore, Hoch et al.’s challenge does not invalidate our use of an economic perspective to explore whether a rationale was published for the establishment of a separate HTA process for cancer drugs.

While the applicability of an economic perspective is not contingent on the goal chosen, the applicability of CEA is. Cost-effectiveness analysis is based on the premise that the goal of decision makers is to maximize the total aggregate health benefit conferred to a population for a given level of resources [3–6]. Hence, we are puzzled by how CEA is being used to inform pCODR recommendations if maximizing health benefits with available resources is not a goal the pCODR intends to achieve.

Finally, if challenging our assumption that maximizing health with available resources is not the goal, then we assume that Hoch et al. know what the pCODR goals are. As such, they should transparently state the goal or goals—an issue we address further in Sect. 2 below.

2 The Suggestion that “Decision Makers may be Pursuing Different Objectives from Academic Researchers”

Hoch et al. state that “other valid rationales do exist” for why Canada has a separate cancer drug HTA process and that “decision makers may be pursuing different objectives from academic researchers.” However, they do not tell us what these other rationales are or what ‘different objectives’ (which we call ‘goals’) decision makers might be pursuing. Instead they suggest that this could be explored through future research.

The pCODR is a publicly funded process for making recommendations regarding the allocation of public drug budget dollars. We do not believe it is good public policy to require stakeholders (which would include the general public (taxpayers), patients, physicians, caregivers, and manufacturers) to make guesses regarding the goal(s) that publicly funded agencies or processes are intended to achieve. Instead, we suggest that transparency is paramount. The goal(s) that the pCODR is intending to achieve, the reason why these goals were chosen, how the pCODR will best achieve these goals with available resources and how the pCODR arrives at its recommendations should all be transparently stated. This transparency would enable stakeholders to assess the extent to which the recommendations can be expected to facilitate achievement of the stated goal(s), whether the criteria used to determine recommendations were applied in a consistent manner and whether a separate reimbursement review process is the best way to achieve the stated goals. In other words, it would facilitate accountability by enabling stakeholders to monitor performance.

The pCODR process has been in operation since 2010 and has been used to make recommendations since 2011. In challenging our paper, we were expecting that Hoch et al., as individuals who are very involved in the process of generating pCODR recommendations, (1) would acknowledge knowing what the rationale and goals of the separate HTA process for cancer drugs (i.e. the pCODR goals) are, and (2) would reveal these goals in their commentary. However, they have not provided any such clarity about the goals to be achieved or the rationale for a separate HTA process for cancer drugs.

Hoch et al. state that “Perhaps 100 % transparency is not [decision makers’] main goal” and suggest that “through observation of their behavior and discussion, it is possible to perceive a revealed preference for a separate cancer drug HTA process. If provinces are assumed to maximize some objective function, then there must be some good that comes from a separate cancer drug HTA process.”

In effect, Hoch et al. want readers to accept that because something is happening, there must be some good coming of it (without providing any evidence and without stating what this ‘good’ is). This is a belief-based system, which requires stakeholders to trust that the pCODR process has clear goal(s), that these goals are the desired goals of society and decision makers, and that the pCODR recommendations are facilitating the best achievement of these goals from available resources. However, we strongly advocate—particularly when public dollars are in question—not for a system of ‘trust us’ but for a system based on transparency and accountability (i.e. being able to check on performance).

While Hoch et al. state that our literature search was not a methodologically robust systematic review, we question why such a systematic review should be required in order to find the rationale and goals for a publicly funded process such as the pCODR. In the spirit of transparency and public accountability, these goals should be in the public domain and easy to find. Furthermore, given that Hoch et al. suggest that “Perhaps 100 % transparency is not [decision makers’] main goal”, the ability of a systematic review to better address the questions we posed in our paper is unclear.

Finally, Hoch et al. note that “We must also face the reality that the purpose of our role may be to promote goals related to process rather than outcome.” Even if there are process-related objectives, there must still be an intended outcome that the process objective is expected to achieve. If there is no clarity regarding the outcome to be achieved when allocating scarce resources, how can stakeholders assess whether a separate process (and process-related objective) is required and represents the best use of limited resources? As Winston Churchill observed many years ago, “However beautiful the strategy, you should occasionally look at the results.”

3 Hoch et al.’s Prescription for How to Make “Positive Contributions that Help Decision Makers Address the Challenges that Decision Makers Have”

Hoch et al. propose two options for health economists who desire to be more involved in HTA policy matters: “(1) explaining to decision makers (e.g. through scientific publications) that they are not behaving as economics dictates they should or (2) studying how we can be of assistance and then attempting to do that.”

We first note that these are not the only options for health economists. Health economists can also help decision makers to achieve their defined resource allocation goals from available resources when decision makers are clear about the goals they want to achieve. In order for health economists to help in this way, as we explained (above), these goals should be clearly stated without any need to guess what they are. When economists infer the goal when it is not clearly stated (e.g. maximizing health benefits with available resources), they run the risk that decision makers will refute that this is the intended goal.

Hoch et al. state that our article “can be viewed as demonstrating the futility of the first option” (above). However, in applying an economic approach, our work raises important questions about the allocation of public dollars to a new and separate HTA review process for cancer drugs, highlighting the importance of transparency and accountability. Furthermore, we were pleased that our work evoked such an impassioned response, as this created a perfect opportunity for Hoch et al. to clarify any misinterpretations that stakeholders may have regarding the rationale and goal(s) for establishing a separate HTA process for cancer drugs. However, Hoch et al. do not provide any further clarity in this regard.

We suggest that the question of how to be of assistance to decision makers applies not only to health economists but also to the pCODR. Using the example of Ontario, Hoch et al. acknowledge that, following provision of the pCODR recommendation to the provinces, the decision maker then “negotiates with the drug company over whether and how the drug will be covered”, stating that “These negotiations happen behind closed doors and are confidential” and that “people who know what happens, cannot tell you (by law), and people who tell you what happens cannot know (by law).” This raises a number of questions. For example, how can Hoch et al. be sure that the pCODR recommendations are helping decision makers with the challenges they have, given that the pCODR may not know what the decision makers ultimately want to achieve? Furthermore, even if the goal(s) of the pCODR and the decision maker were known and identical, how can Hoch et al. be sure that the pCODR recommendations are helping to achieve the goal(s) given that the ultimate decision is negotiated “behind closed doors”?

Interestingly, Hoch et al. state that “Additionally, because decisions are made behind closed doors and recommendations are not, the advice one hears about what is useful is from an academic or recommendation point of view, but it is usually not from a decision-making point of view.” We are confused by this statement. Hoch et al. appear to be challenging our work because, they suggest, it makes recommendations without knowing/understanding what decision makers want. However, if the pCODR recommendations are made from a “recommendation point of view” and the decisions that are “made behind closed doors” are made from a “decision-making point of view”, then it would appear that the pCODR is also making recommendations without knowing/understanding what the decision makers ultimately want.

4 Conclusion

In our paper [2], we explained why an economic perspective is an appropriate perspective to use in assessing whether a rationale has been published to explain the establishment of a separate drug reimbursement review process for cancer drugs. In reading Hoch et al.’s commentary, we are confused as to whether they truly think that an economic rationale is not needed (given that they describe the task facing decision makers in economic terms), whether they disagree with the inferred goal of maximizing the health benefits to the population with available resources, or both. What is particularly concerning to us is that the authors appear to argue that having a separate HTA process for cancer drugs in Canada is a good thing (even claiming that “there must be some good that comes from a separate cancer drug HTA process”) but fail to define the rationale and goal(s) that are intended to be achieved via this separate process and the ‘good’ (which we would expect to be in the form of achieving these goals) that is coming from it. Consequently, it is not clear to us what stakeholders are supposed to base their acceptance of the suggestion that the separate system is ‘good’ on. As we discuss above, it is easier to judge whether or not something is ‘good’ when the goals that are intended to be achieved via the allocation of resources, the reason why these goals were chosen and the way in which the stated goals will be achieved are clearly and transparently stated.

Hoch et al. dismiss our work, citing the importance of “getting the question of interest right”, proposing that researchers need to “consider a broader view of what decision makers want” and suggesting ways for research to be ‘useful’. We are surprised that research that seeks to identify the rationale behind a decision regarding the allocation of public dollars and that endorses the principles of transparency and accountability regarding such resource allocation decisions is not seen as helpful or useful. We challenge the implication that work is not ‘useful’, ‘positive’ or ‘helpful’ simply because its conclusions are incongruent with the opinions of other researchers, and we strongly suggest that dismissing research simply because it raises questions about the status quo is a dangerous approach to public policy dialogue.