Background

Baltussen and others have proposed that multi-criteria decision analysis (MCDA) can improve priority setting processes [1,2,3]. Other approaches include cost-effectiveness analysis [4, 5], criteria-based systems that include cost-effectiveness and other concerns [6,7,8,9], programme budgeting and marginal analysis (PBMA) [10], procedural frameworks such as accountability for reasonableness [11], and rights-based approaches [12].

For example, the World Health Organization and the World Bank have championed cost-effectiveness as a key criterion for global and national priority setting [4, 5]. In UK, the National Institute for Health and Care Excellence (NICE) identifies the most cost-effective services through health technology assessment, with input on key values from the Citizen’s Council. In Thailand, the Health Intervention and Technology Assessment Program (HITAP) appraises a wide range of health technologies and public health programs by six criteria: size of population affected, severity of disease, effectiveness of health intervention, variation in practice, economic impact on household expenditure, and equity and social implications [8, 9].

Common for all priority setting approaches is that they seek to provide a ranking of services delivered at different levels of the health system by linking evidence on needs and outcomes with values, principles, and criteria that have support in the population. The approaches differ along at least two axes: (1) technical versus deliberative, and (2) narrow focus versus comprehensive.

MCDA as discussed in this issue of the journal have moved from a more technical framework with a narrow focus towards a deliberative framework allowing for discussion and use of a wide range of criteria and concerns [3]. In my view, MCDA has moved too far from its origin. In the further development of MCDA tools, four critical questions therefore need further scrutiny.

Technical or deliberative approaches?

Already in 1998, the ethicist Soren Holm said “goodbye to the simple solutions”, arguing that the false dichotomy between substantive and deliberative approaches to priority setting would be replaced by a third phase of integration [6]. A similar development may be seen for MCDA. If both quantification and deliberation is key, how can the two approaches be integrated? Enthusiasm for deliberation should not lead to the abandonment of quantification.

How many criteria can a decision-maker handle?

The trend of embracing all possible criteria for priority-setting analysis is, in my view, unfortunate. While moving beyond cost-effectiveness as a single criterion is appropriate, quantitative models may not be able to handle an unlimited number of additional concerns without ending up as another black-box approach. Likewise, stakeholders discussing priority setting decisions may not be able to handle and balance 10–15 different considerations for a large set of interventions or services at the same time. Handling such large amount of data is simply not possible in a comprehensive way. The field needs to set priorities among the criteria included in the analytical framework. I am prepared to argue that, at the most general level, we need only four criteria: level and distribution of health outcomes; level and distribution of non-health outcomes.

Quantitative approaches: additive or multiplicative weights?

Current practice in MCDA for priority setting is to ask key stakeholders which criteria they find relevant and ask them (through discrete choice experiments) to assign weights to each criterion. The MCDA analyst then rank services based on stakeholder values. Often additive weights are used; and sometimes 10–15 criteria are identified in this way. If, say cost-effectiveness is assigned a weight of “20%” though this process (and the remaining 80% is distributed to the other criteria), the result strikes me as problematic. What does “20%” mean? I believe the use of additive weights is at risk of forcing key stakeholders to end up with a rank-order that is different from what they would end up with simply through individual judgment or a purely deliberative process. One possible explanation is the choice of weighing scheme.

Consider this example (Table 1). Three interventions have the same cost, different benefits, and target groups with different incomes. Let us assume that stakeholders care about only two concerns: equity and benefit maximization.

Table 1 Three interventions (a–c) with the same cost, different benefits, and target groups with different incomes (PW poverty-weighted)
Full size table

In A, stakeholders think that benefits to the poor have three times the value of benefits to the rich; in B, stakeholders also think that benefits to the poor are more important than benefits to the rich, and they assign equal weight to ‘equity’ (50%) and ‘benefit’ (50%).

As can be seen from the table, using multiplicative weights would clearly rank the three interventions, while using additive weights would not—in this example. The reason is that additive weights, as used in many forms of MCDA, are applied to ordinal ranks. There is loss of information compared to the multiplicative weighing scheme. Methods for assigning weights to different concerns and their rationale need to be further discussed and explored in MCDA.

Deliberative approaches: who are invited to the table?

For deliberative approaches to MCDA, issues related to representation and power asymmetries require further clarification [13]. The outcome of MCDA depends crucially on who the participants are. Current practice tends to favor health policy decision-makers and health planners, with some patient or user-participation. Given the technical nature of priority setting, there may be feasibility constraints on who can participate. Yet, this foundational issue for approaches that aim to be more democratic is insufficiently explored. Who is invited to the table matters.

Conclusion

In summary, recent developments of MCDA tools for priority setting are detached from their theoretical foundations: deliberation without proper quantification can mislead; choosing too may criteria can confuse; using additive weights causes loss of information, and the choice of participants is arbitrary.