Cost-Effectiveness in Global Surgery: Pearls, Pitfalls, and a Checklist

Abstract

Introduction

Cost-effectiveness analysis can be a powerful policy-making tool. In the two decades since the first cost-effectiveness analyses in global surgery, the methodology has established the cost-effectiveness of many types of surgery in low- and middle-income countries (LMICs). However, with the crescendo of cost-effectiveness analyses in global surgery has come vast disparities in methodology, with only 15% of studies adhering to published guidelines. This has led to results that have varied up to 150-fold.

Methods

The theoretical basis, common pitfalls, and guidelines-based recommendations for cost-effectiveness analyses are reviewed, and a checklist to be used for cost-effectiveness analyses in global surgery is created.

Results

Common pitfalls in global surgery cost-effectiveness analyses fall into five categories: the analytic perspective, cost measurement, effectiveness measurement, probability estimation, valuation of the counterfactual, and heterogeneity and uncertainty. These are reviewed in turn, and a checklist to avoid these pitfalls is developed.

Conclusion

Cost-effectiveness analyses, when done rigorously, can be very useful for the development of efficient surgical systems in LMICs. This review highlights the common pitfalls in these analyses and methods to avoid these pitfalls.

Appendix

The theoretical basis of the ICER

A more thorough discussion of the difference between the “shopping spree” and the “competing choice” problems follows.

The “shopping spree” problem [45] is answered by what Murray et al. [47] have called “generalized CEA.” In this problem, the decision-maker begins with a blank slate. No health system exists—it will be created from scratch. The decision-maker is at a hypothetical health system “shopping mall”, at which she gets to decide which of a number of non-exclusive types of interventions she should include in her shopping cart. Should she treat HIV? Should she include surgery? Should she provide vaccinations against human papilloma virus? Should she perform mammography? All sorts of options are available to her; her only constraint (but it is an important one) is that of her budget.

This sort of an analysis maximizes “allocative efficiency” [47]—that is, the money available to the decision-maker is spent in such a way as to maximize the benefit she is able to buy with it.

This may be best understood with a toy example. Assume the decision-maker is not creating a health system from scratch but shopping for her apartment. She has $10 to spend and has the following options at her disposal (each listed with its cost and the amount of “happiness”—in a unitless measure—it affords her. Assume for the purposes of this example that she knows both the cost and the happiness with certainty):

Item	Cost	Happiness	C/H
Toilet paper	$2	2	$1
Cereal	$4	3	$1.33
DVD	$10	5	$2
Wine	$15	3	$5
Ice cream	$4	10	$0.40

The final column of this table is the decision-maker’s cost-per-happiness ratio. She should obviously spend her first $4 on ice cream, because this nets her the most happiness per dollar. Next most efficient is toilet paper, followed by cereal.

After purchasing these three items, she hits her budget constraint. Having spent all her money, she has gained 15 happiness points—the most she could gain with $10 (we leave it to the reader to convince themselves that there is no other combination of goods she can buy that will net her more than 15 happiness points for $10).

At this point, we can conclude that the decision-maker’s “willingness-to-pay” ratio (how much she would pay for one unit of happiness) is $1.33, because that is the most she was able to pay for her last unit of happiness.

The translation into the allocatively efficient design of a healthcare system is obvious. However, we want to draw attention to the two settings in which this sort of decision-making occurs: It either occurs in the de novo design of a healthcare system or when a decision-maker has found herself with a sudden increase in her budget constraint. In settings in which neither of these two conditions holds, the shopping spree problem is irrelevant.

We also want to draw attention to the metric used in decision-making for the “shopping spree” problem. Instead of the ICER in Eq. (1), the shopping spree problem is answered by comparing simple cost-effectiveness ratios. This is because, with an empty shopping cart, the counterfactual in the ICER is “nothing”. To be explicit:

$${\text{ICER}} = \frac{{c_{\text{a}} - 0}}{{e_{\text{a}} - 0}} = \frac{{c_{\text{a}} }}{{e_{\text{a}} }}$$

The “competing choice problem” [47] is answered by what Murray et al. [47], have called “intervention-mix-constrained CEA”. This sort of problem presents itself when the health system is already up and running, the budget constraint has not changed, and a new intervention that is mutually exclusive from an intervention already in the health system presents itself for evaluation. In this case, an ICER is appropriate, as will be seen.

To return to the previous example. A competing choice problem exists when the decision-maker, having filled her shopping cart, encounters a second brand of toilet paper. This new toilet paper offers more happiness (2.5), but at a higher cost ($3). Should she swap out her old toilet paper for this new brand?

If the decision-maker uses the simple (non-incremental) cost-effectiveness ratio, she will make the wrong decision. The ratio for this new toilet paper appears favorable ($1.20, less than the $1.33 she is willing to pay), so, at first blush, it appears she should swap her old toilet paper.

However, with a budget constraint of $10, buying the new toilet paper decreases her overall happiness to 14.75.

What happened?

By using a simple cost-effectiveness ratio, she answered the wrong question. She already has toilet paper in her shopping cart—the question is not whether she should add this second brand to the already full cart (doing so would leave her very little money left for cereal), but whether she should swap the old toilet paper for the new. The two brands are competing choices.

It is more correct to ask, “Is the added happiness she gets worth the added cost she would have to pay?” That is, is the incremental benefit she gains from the substitution worth it to her. An ICER is appropriate:

$${\text{ICER}} = \frac{\$ 3 - \$ 2}{2.5 - 2} = \$ 2$$

In this case, she would be effectively paying $2 for the additional happiness she gets from the new toilet paper. This is will above her willingness-to-pay ratio of $1.33; she clearly should keep her original basket of goods.

Again, the translation to global surgery is evident. As has been discussed in the main text of the paper, a CEA of a mission trip to perform hernia surgeries is not an analysis of hernia surgery in general, but an analysis of a mission trip to perform hernia surgeries. Unless the health system in the country of interest performs no hernia surgeries whatsoever, using simple cost-effectiveness ratios to evaluate this mission trip will vastly overstate its cost-effectiveness, when money may truly be better allocated improving the national surgical system instead.

Comparing multiple interventions

The ICER may be used to compare multiple mutually exclusive interventions. As a hypothetical example, we imagine three options for the treatment of thyroid diseases in a target country. The first is the status quo, which is hemithyroidectomy alone. The second couples subtotal thyroidectomy with short-term calcium supplementation, while the third is total thyroidectomy with both calcium supplementation and thyroid hormone replacement.

Intervention	Cost	Incremental cost	Effectiveness (DALYs averted)	Incremental effectiveness	ICER ($/DALY averted)
Hemithyroidectomy	$1000	–	0.8	–
Subtotal thyroidectomy	$1500	$500	0.9	0.1	$5000
Total thyroidectomy	$2700	$1200	0.95	0.05	$24,000

1. The numbers in this table are fabricated and are used only for the purposes of this example

We have ordered the interventions from least expensive to most expensive (and, incidentally, from least effective to most effective). The ICER for the subtotal thyroidectomy option, compared against the next most expensive option, is

$${\text{ICER}} = \frac{\$ 1500 - \$ 1000}{0.9 - 0.8} = \frac{\$ 500}{0.1} = \$ 5000/{\text{DALY}}\;{\text{averted}}$$

Similarly, the ICER for the total thyroidectomy strategy is $24,000 per DALY averted. In a country willing to pay up to $10,000 per DALY averted, subtotal thyroidectomy is the correct option. For countries (as in the USA) willing to pay more than $24,000, total thyroidectomy is the correct option.

Of note, ordering by increasing cost does not necessarily guarantee an ordering by increased effectiveness:

Intervention	Cost	Incremental cost	Effectiveness (DALYs averted)	Incremental effectiveness	ICER ($/DALY averted)
Hemithyroidectomy	$1000	–	0.8	–
Subtotal thyroidectomy	$1500	$500	0.75	–0.05	N/A
Total thyroidectomy	$2700	$1700	0.95	0.15	$11,333

In this hypothetical case, subtotal thyroidectomy is more expensive and less effective than hemithyroidectomy. An ICER is meaningless in this situation because subtotal thyroidectomy should not be chosen—for less money, the health system could get more benefit by simply doing hemithyroidectomies. Subtotal thyroidectomy is said to be “dominated” by hemithyroidectomy. As such, the ICER calculation is just between the two non-dominated options: hemithyroidectomy and total thyroidectomy.