1 Firearms and Freedom

The Wisconsin Department of Natural Resources is reported as having issued more than 615,000 gun licences for the 9-day hunting season in 2013Footnote 1, getting on for twice the number of US Army personnel deployed at the peak of the Vietnam War in 1969Footnote 2. Although every hunting season sees its share of accidents, injuries and deaths, it is probably safe to assume that participants continue to favour their long-held venerated right to bear arms. Gun control is not going to be a vote winner amongst these citizens any time soon. Remedies have been sought by the aggrieved victims of deliberate or accidental gun-related harm across the USA—class actions against the industry, law suits seeking compensation from individual manufacturers and dealers—often times meeting a well-rehearsed rebuttal based on the defence that firearms technology is essentially safe, and the problem, if there is one, is its misuse by those who access it. There is a remarkable degree of similarity between the backwoodsmen of the mid-West and hardcore health economists back East. No matter where you stand on the issue, let us park those ideas for the time being.

Economic evaluation of large-scale public spending programmes received the tacit endorsement of government agencies in the USA and UK as long ago as the 1960s. Whether it was investment in space exploration or the development of infrastructure for air travel, the fundamental question emerged as to whether the value of the benefits outweighed the costs. From its very inception, the science of health economics has grappled with a problem that remains unresolved to this day, namely the issue of the valuation of health benefits. Whilst costs are typically measured in terms of monetary units, there is no standard metric that can/should be applied when quantifying the ‘value’ of benefits. Indeed, different forms of economic analysis require different outcome metrics, hence cost-utility analysis—a specific form of cost-effectiveness analysis—relies upon an operationalised concept that is formed by the arithmetic product of quality and quantity of life. Whilst the quality-adjusted life-year (QALY) has a distinctive role in economic evaluation, it has little traction in any other setting. It is this representation and measurement of health benefits that poses the greatest challenge for health economists and citizens alike.

2 Measuring Benefits

Cost effectiveness, like apple pie and motherhood, appears to be an unassailable positive virtue for any enterprise. To deny this would be to implicitly accept that neither cost nor effectiveness need be a consideration in healthcare or any other realm of individual or public experience. However, once consolidated into high-level decision-making processes, as with the UK National Institute for Health and Care Excellence (NICE), where cost effectiveness contributes directly to the determination of whether new interventions should be made available to the population at large, then it is incumbent on all parties to ensure that it has demonstrable legitimacy. Failure to achieve such an endorsement in fact, or even to be motivated to do so, is at heart of concerns that cost-effectiveness analysis as currently espoused and practiced risks compromising scientific principles and thereby undermines claims that it is an essential component in decision-making processes as applied to healthcare. Worse still, it may act as a technocratic smokescreen that separates the citizen and his/her welfare from those charged with protection of it.

At the heart of the problem is the fact that whilst the concept of the QALY is elegantly simple, its representation as a cardinal measure of health rests upon more complex components. It requires that we know how long individuals survive in a given health state AND that we know by how much to adjust the weight/value/importance of that time. It is this latter aspect of the QALY calculation where health economists have been granted the widest licence to operate and it is here too that there is (or should be) the greatest cause for concern regarding the erosion of due process. It might be supposed that in thinking about how we might value the quality of our survival, we might wish to consult widely amongst the many sources of opinion that could inform that judgement. In fact, the task has been acquired by health economists largely by default, as a result of their being the progenitors of the QALY enterprise. The fact that non-(health) economists were excluded from the process of deciding how to value health might not be so troubling were it not that the cost-effectiveness evidence based on QALYs is used to inform healthcare decisions that affect significant proportions of any country’s ordinary citizens. So the method chosen to value health is determined by a select group of technicians entrusted with that task. Nothing new in this, one might think—we have specialists who advise governments around the world on matters that require their expertise and on subjects that are largely opaque to public viewing. So perhaps there is a case for allowing health economists to determine the means by which we value the health benefits for use in cost-effectiveness analysis for publically funded healthcare? Such a step would be less controversial if it involved at least two other considerations. First, that the decision is publicly deliberated upon prior to enactment, giving an airing to all stakeholder voices. It would emphasise too, the case for a wider debate about measuring values more generally—a political minefield that the health economist has largely been able to skirt around until now. The second requirement for entrusting the valuation task to health economists is that they are able to collectively demonstrate a consensus as to the method by which health benefits should be measured in practice. It is this requirement that has been crucially omitted and that opens up the science and its practitioners to ridicule.

3 f (utility)

Agreement amongst health economists on this vexed question of how to measure the value of health benefits is further mired by the concept of ‘utility’. There is a widely held view that the quality-adjustment factor used in QALY calculations should be based on utility weights. Leaving aside for the moment the vexed question as to whose utilities/weights should count, it is pertinent to enquire as to exactly how such weights are to be established. This then brings us to the very heart of the matter, since health economists have yet to agree upon any single means of doing so. Worse still, they appear content to accept the co-existence of multiple procedures for measuring utility whilst at the same time advising NICE and other regulatory agencies to adopt guidelines that are patently dishonest. Historically within the wider health economics fraternity, two utility elicitation procedures have been recognised—standard gamble (SG) and time trade-off (TTO)—the latter being developed as a surrogate for the former. More recently other procedures have been proclaimed to be at least their equal if not indeed preferable, for example discrete-choice experiments (DCEs). What is most striking here is that there is general acceptance that SG and TTO yield systematically different results when used to estimate the utility of a given health state. Furthermore, there is no transformation function that allows us to convert TTO utilities into SG utilities; this is in marked contrast to other situations where multiple metrics co-exist, as with, say, measurement of temperature in degrees Celsius and Fahrenheit. Utilities based on TTO and SG are non-commensurate and cannot be substituted one for the other. It follows that QALY calculations based on utilities estimated by these two inimicable methods are likely to differ, as may the incremental QALY gains resulting when one intervention is compared with a second. The magnitude of a cost-effectiveness ratio can be disproportionately influenced by the arithmetic value of the denominator. An incremental cost of (say) $5,000 lies on one side of a 50,000 cost/QALY threshold if the QALY gain is 0.101 and on the other side if the gain is 0.098. Small differences in QALY gains are highly likely to emerge if different utility elicitation procedures are employed. Of course, that might be less of an issue were it the case that a single procedure was recognised as being the standard. But no such principled position has, in fact, been adopted.

4 Papering Over the Cracks

There is an old adage found in many countries that a man with one watch always knows what time it is, but a man with two is never sure. So it is with utility measurement. What makes this absurd situation so deeply troubling is that health economists when faced with this self-evident truth that they do not in fact have a monopoly of certainty regarding the valuation of health will admit that yes indeed, there is some doubt about whether TTO or SG represents the ‘true’ social value of health and that furthermore, both could be in error. Yet, despite this state of affairs, they persist in requiring that cost-effectiveness evaluation must be conducted according to the rules that they specify—including the frank nonsense about the need for utility weights in calculating QALYs. In response to critical comments about what probably constitutes scientific malpractice, adherents of the cost-effectiveness persuasion point to the need to make decisions. “No decision is still a decision” they will remind us. But what if such decisions are ill informed, what if marginal QALY benefits are misrepresented? We are to be placated apparently because probabilistic sensitivity analysis has shown (after the event) that the quality-of-life estimates in the models used to determine cost effectiveness were tested to destruction and there was little or no impact on the incremental cost-effectiveness ratio (ICER) or side of the threshold on which it was located. So that’s alright then, and we may sleep soundly in our beds. But if that is a universal state of affairs, then why do agencies such as NICE insist that social preference weights are based on TTO—especially since those weights are inconsistently included in cost-effectiveness submissions made to them? We can be further reassured that the right decisions are being made, at least as far as the cost-effectiveness information is concerned, since UK health economists seem content to adopt the argument that NICE utilises the “best available” evidence. But what if that evidence is tainted? “Don’t let the perfect be the enemy of the good” will be the rejoinder. Sticking to the principle that health economics should be governed by the same ethical standards that apply in other areas of science, this laid-back attitude ought really to worry those who advocate for an extended role for cost-effectiveness analysis in healthcare. It is rather like giving untrained citizens access to firearms without proper safeguards and then being surprised that things sometimes/often take a turn for the worse. The victims of the abuse of firearms can mostly be identified. Those adversely affected by decisions influenced by poor-quality cost-effectiveness analysis have no such clarity of name or number.