FormalPara Key Points for Decision Makers

The ‘societal perspective’ has been advocated, but less consideration has been given to what this should include and its practical implementation.

This paper presents a framework that can inform multiple decision makers.

The framework sets out the assessments to be made and distinguishes points at which value judgements feed into the evaluation.

1 Introduction

In the context of a free market, individuals are the decision makers who determine their own resource allocation. However, in most societies, a proportion of available resources is allocated by the government through some form of collective decision-making process [1,2,3]. The entities of the government, each with a focus on a particular part of the public sector, for example, health, education and criminal justice ministries, therefore, represent another set of decision makers who determine the allocation of collective resources [4]. The government also provides a mechanism for constraining the choice set of individuals [1]. Thus, resource allocation is accomplished through a mix of market forces and the agency of the government [1,2,3]. Institutional arrangements constrain the choice set, and, subject to these, social choices must be made involving the allocation of collective resources for the provision of goods and services in the public sector [2].

The methods of economic evaluation have developed to inform collective decisions by examining the resource requirements and outcomes of alternative policies or interventions [5]. Identifying policies that could have social value involves a series of questions of value and of fact.Footnote 1 Normative questions of value determine the outcomes relevant to inform a decision, their relative worth, and judgements about their desired distribution in the population. Key normative judgements needed ex ante include defining the outcomes that will influence the choices made by decision makers. Subsequently, economic evaluation can proceed as a factual (empirical) account regarding what would change if resources were allocated to that intervention instead of alternative options. The introduction of a new healthcare intervention that is more costly than current practice results in opportunity costs. In the case of fixed budgets, these take the form of the benefits of displaced activities that can no longer be funded. When budgets are more flexible, opportunity costs relate to the benefits associated with the broader set of activities to which the resources could have been devoted. In other words, the opportunity costs of new interventions relate to other activities within a sector unless budgets are increased to fund more costly activities. Even when budgets adjust immediately, opportunity costs exist somewhere as the resources could have been used for other purposes [6].

Hence, the changes in outcomes attributed to the new intervention must be compared with those opportunity costs. Once the outcomes that are gained and forgone are estimated, normative judgements about their relative worth and desired distribution across individuals may be crucial to inform trade-offs and resource-allocation decisions, for example, when some individuals gain and others lose.

For some interventions (e.g. some medical technologies), the majority of impacts fall on one sector (e.g. the healthcare sector), and the minor, wider effects are often assumed unimportant and excluded from the scope of the evaluation. However, many interventions have important effects on costs and outcomes that fall across the private and public sectors or between different entities within those sectors. Here resource-allocation decisions that reflect the full range of effects may need input from multiple decision makers who may have different objectives and remits. For example, an intervention may include decision-maker involvement from both healthcare and education sectors. Similar issues arise when considering the need for coordination between different levels of decision makers within a sector, such as at national and local levels. To evaluate such interventions, the use of a catch-all ‘societal’ perspective for economic evaluation has often been advocated, whereby all costs and outcomes are reflected [7,8,9]. However, many studies do not take a societal perspective, and even those that state they do omit potentially important costs and outcomes [10,11,12,13,14]. Even when a societal perspective is undertaken that captures all costs and outcomes, it is unclear how the summary information produced by such an approach informs choices across different settings and decision makers [15]. Different objectives may lead to different judgements about what outcomes are relevant and their relative values. The low proportion of studies that even attempt to incorporate a wider perspective may be a consequence of these challenges [11]. However, use of a narrow perspective tailored to one decision maker risks omitting important outcomes for other decision makers.

The framework proposed in this paper describes how economic evaluation can inform multiple, heterogeneous decision makers and provide guidance for an overall societal perspective, which has been missing in the economic evaluation literature. To do so, it clearly distinguishes the points at which value judgements feed into the evaluation and the implications of alternative judgements with respect to the final results. The logic applies to a range of settings, including where budgets and resource-allocation decisions are determined simultaneously and/or where different decision makers are seen as having different objectives and remits.

The paper expands on the ‘impact inventory’ of the Second Panel on Cost-Effectiveness in Health and Medicine [10] to capture the impact of an intervention on individuals from a set of outcomes or dimensions of interest determined by value judgements and institutional arrangements of the decision makers to be informed. The inventory catalogues the impacts on these outcomes in terms of both the direct effects and the opportunity costs and forces the analyst to set out explicitly what outcomes and sectors are included in their analysis. Alternative approaches for aggregating the impacts are then considered. Using a case study from the Second Panel, a series of assessments is set out for evaluating an intervention involving a range of public sector decision makers contingent on institutional arrangements. The approach is then compared with that proposed by the Second Panel to expose the implicit judgements and potential weaknesses in their societal perspective approach. Finally, a general discussion is presented of the economic theory and underpinning value judgements that support different approaches, and the role for economic evaluation.

2 Extended impact inventory framework for economic evaluation

Economic evaluation of interventions generally starts from impacts on individuals, which could be through changes in any number of dimensions relevant to them (e.g. impacts intrinsic to the individual, such as aspects of human capital [health, education], their capabilities or changes to commodities they consume). In the impact inventory’s purest form, all possible dimensions could be included, each captured in natural units which include no intrinsic value judgements. However, in practice, it is unlikely to be possible to be exhaustive in this selection process, and the choice of the dimensions to be included is a normative judgement. Each public sector decision maker will have in mind their objectives and the key issues of consequence when assessing the value of an intervention. To inform decisions appropriately, the dimensions in the impact inventory should reflect the key issues of consequence for any decision makers it seeks to inform, for example, healthcare decision makers may be interested in the health of individuals. In some instances, the decision maker’s objective will not align with an obvious measure, in which case their preferred choice of constructed or proxy measure can be elicited. The identification of the relevant dimensions is considered further in Sect. 3.

Once the dimensions are defined, the evaluation starts by measuring the changes in these dimensions for each individual potentially affected (i.e. those whose dimensions are expected to change as a result of the introduction of an intervention). For this there are two parts: (1) the direct effects of the intervention and (2) the opportunity costs in terms of what individuals would otherwise achieve from the alternative use of the resources (what is forgone). The absolute level of a dimension may also be important (e.g. as a result of equity concerns or diminishing marginal impact), in which case the impact inventory can also include the current allocation. Populating each cell of the inventory is a question of fact: in principle, knowable from evidence for each individual and dimension.

Figure 1 represents a generic impact inventory with x dimensions (D) and n individuals (P) in terms of current allocations (CA), direct effects (DE), opportunity costs (OC) and net effects (\(\Delta\)).Footnote 2

Fig. 1
figure 1

Impact inventory reflecting the direct effects and opportunity costs across all dimensions for each individual. CA current allocation, Dj dimension j, DE direct effect, Pi individual or group i, OC opportunity cost, ∆ net effect

Full size image

If the new intervention is introduced, individual Pi would gain DEij in dimension Dj directly. However, if the intervention had not been introduced, they would have gained OCij, from the alternative (which is not funded given the introduction of the intervention). The net effect on each dimension, \(\Delta_{ij}\), is the difference between the direct effect and the opportunity cost (e.g. \(\Delta_{ij} = {\text{DE}}_{ij} - {\text{OC}}_{ij}\)). Assuming all dimensions are characterised as ‘positives’, i.e. more is better for all individuals, then where the new intervention leads to a net gain in at least one dimension for one individual and no net losses in any dimension for any individual, this intervention could be described as beneficial (no-one is worse off and at least one individual is better off). However, in most cases, there will be gains and losses both within individuals (i.e. some dimensions improve while other worsen) and between individuals (i.e. some individuals will gain overall, and others will lose). Therefore, to judge whether introducing an intervention leads to a net gain overall, methods for aggregating across dimensions and individuals are required.

There are two general aggregation approaches: (1) a within-individual approach (to aggregate first within individuals across all dimensions, and then to aggregate across individuals) or alternatively (2) a within-dimension approach (to aggregate first across individuals for each dimension, and then to aggregate across dimensions at the population level).

2.1 A Within-Individual Approach

To implement a within-individual approach, a benefit function must be specified to aggregate across dimensions for each individual. Many specifications are possible, each functional form (F) representing a normative judgement on how dimensions are valued at the individual level. A common function could be based on a representative individual (i.e. F is assumed the same across all individuals) or multiple functions could account for heterogeneity across individuals, e.g. as a result of differences in preferences (Fi). For example, a common aggregate function could be based on relative values at the margin as determined by market prices.Footnote 3 An alternative would be to aggregate dimensions based on relative valuations elicited from a sample of the public, relevant decision makers or other relevant experts. The person-level (P) net benefit function for each individual (i) can be generally specified as:

$${\text{NB}}_{Pi} = F_{i} \left( {\Delta_{i1} ,{\text{CA}}_{i1} , \ldots ,\Delta_{ix} ,{\text{CA}}_{ix} } \right)$$
(1)

This represents the net benefit of the intervention to the individual, that is, the benefit to the individual from the intervention being introduced less the benefit to the individual if it had not been introduced. The inclusion of the current allocation of each dimension captures only the value of the allocation for the individual, not the interpersonal or distributional value across individuals (i.e. it does not allow for valuing the outcomes of other individuals in this simple case). With a function specified, evidence of the impact of an intervention can be aggregated, which allows estimation of the overall net benefit to each individual of the intervention.

If the intervention results in a negative net benefit for any individual (and positive for others), an overall population net benefit function is required that aggregates across each individual. As with the individual net benefit function, many specifications are possible, reflecting normative judgements about how the impacts on different individuals are valued. For example, all individuals could be valued equally and the individual net benefit functions simply summed. Alternatively, other concerns could be incorporated, such as equity, with individuals receiving weights determined by equity-relevant characteristics such as their overall benefit.

The net benefit function across individuals can be generally specified as:

$${\text{NB}}_{\text{SWI}} = {\text{S}}\left( {{\text{NB}}_{P1} , \ldots ,{\text{NB}}_{Pn} } \right),$$
(2)

where SWI denotes that the net benefit function is at a societal level (S) and based on a within-individual (WI) approach. Figure 2 shows an impact inventory for a within individual approach.

Fig. 2
figure 2

Impact inventory for a within-individual approach. The information required to populate individual 1’s net benefit function is highlighted in bold and the information required to populate the population net benefit function is shown in the shaded right-hand column. CA current allocation, Dj dimension j, DE direct effect, Pi individual or group i, OC opportunity cost, ∆ net effect

Full size image

2.2 A Within-Dimension Approach

The within-dimension approach first aggregates the impact on each dimension across individuals by specification of a benefit function at the dimension level. Again, many functional forms are possible, representing alternative normative judgements. For example, it may be desirable to sum the unweighted changes in dimensions across individuals, or the changes could be weighted by their current allocations because of diminishing returns or equity concerns. Alternatively, the function could be based on dominance, so that a net benefit will only occur if no individual is made worse off and at least one is made better off. It is not the role of the analyst to define the aggregation function; it should reflect the value judgements of the decision maker(s) involved.

The general form for the net benefit function for dimension j is:

$${\text{NB}}_{Dj} = {\text{S}}_{j} \left( {\Delta_{1j} ,{\text{CA}}_{1j} , \ldots ,\Delta_{nj} ,{\text{CA}}_{nj} } \right).$$
(3)

This represents the net benefit on the dimension (i.e. the benefit on the dimension from the intervention being introduced less the opportunity cost on that dimension). Unless an intervention generates a net benefit, or at least no loss, for every dimension, an overall population net benefit function is required to aggregate across each dimension so that the relative value of each dimension can be considered.

The net benefit function across dimensions can be generally specified as:

$${\text{NB}}_{\text{SWD}} = {\text{F}}\left( {{\text{NB}}_{1} , \ldots ,{\text{NB}}_{x} } \right),$$
(4)

where SWD denotes that the net benefit function is at a societal level (S) and based on a within-dimension (WD) approach. Figure 3 shows an impact inventory for a within-dimension approach.

Fig. 3
figure 3

Impact inventory for a within-dimension approach. The information required to populate dimension 1’s net benefit function is highlighted in bold and the information required for the population net benefit function is shown in the shaded bottom row. CA current allocation, Dj dimension j, DE direct effect, NBDj net benefit dimension j, OC opportunity cost, Pi individual or group i, ∆ net effect

Full size image

2.3 Further Considerations

In the simplest case, whereby the functions are linear and additive (and with common parameters), the within-individual and within-dimension approaches have the same results. However, where the functions are non-linear, the overall net benefit of an intervention will differ, not only according to the net benefit functional form but also according to the ordering by which individuals and dimensions are aggregated. Therefore, careful consideration should be given to the appropriateness of the approach chosen given the requirements of the decision makers being informed.

Individuals’ current allocations may be important in either approach. There may be diminishing returns such that the benefit received, for example, from each additional year of life, might diminish; or there may be equity concerns, such as a greater social value being placed on outcomes to individuals who have relatively less compared with those who have relatively more. The inclusion of current allocations and other individual characteristics will increase the informational requirements to populate the inventory and the complexity of the functional form of the net benefit functions.

A further issue is the independence of the different dimensions. For example, where decision makers’ objective(s) do not align with an obvious natural unit such that a constructed or proxy measure is used, it is possible that different dimensions in the inventory may not be conceptually independent, i.e. they could capture some of the same benefits. For example, the quality-adjusted life-year (QALY), widely used in the economic evaluation of health, has significant overlap with the ASCOT measure, which is used to evaluate social care interventions [16]. In such cases, there is a risk of double counting, which analysts may seek to mitigate through either the choice of the dimensions or with adjustments to the net benefit functions. If dimensions are not independent because statistical or causal relationships exist, then this may invalidate some aggregation approaches.

It may not be possible, or desirable, to express explicit aggregation functions if there are competing views of what determines social value. Where an explicit, complete and coherent view of what determines social value is not possible, the analyst can present alternative values and show the thresholds at which decisions will change. For example, it may be possible to identify the minimal set of value judgements required to establish positive overall benefit. Similar approaches have been used in distributional cost-effectiveness analysis when considering interventions where there are conflicting effects on effectiveness and equity [17]. The steps for developing and implementing the framework are shown in Fig. 4.

Fig. 4
figure 4

The steps necessary to develop and implement the framework based on the impact inventory

Full size image

3 Implementing the Impact Inventory for Decisions Involving the Public and Private Sectors

This section considers how the impact inventory could be implemented to inform different decision makers in both the public and the private sectors. It reflects institutional arrangements common in many countries, evaluative approaches already taken to inform decisions and typically available evidence. In this illustration, the common institutional arrangements are presumed whereby budgets are determined separately from decisions about the funding of particular interventions and services and reflect a political process rather than an explicit consideration of individual preferences. We also assume resource-allocation decisions may be needed from multiple decision makers who may have different objectives reflecting their roles and remits. Given these institutional arrangements, a within-dimension aggregation approach may be most suitable because a within-individual approach would require all decision makers involved to agree to aggregation functions that value all dimensions (beyond those within their remit). A case study evaluating treatments for individuals with alcohol use disorders (AUDs) from the Second Panel is used to demonstrate the impact inventory [10]. Further worked numerical examples are also provided in Appendix A1 in the Electronic Supplementary Material (ESM), and algebraic notation for the impact inventory is provided in Appendix A2 in the ESM.

Using healthcare as an example, the following are considered: (1) the choice of dimensions of interest, (2) how opportunity costs can be estimated and (3) the appropriate methods for aggregating outcomes to assess value. The application is then expanded beyond healthcare to also consider impacts on individuals’ private consumption. Finally, an intervention is considered that also impacts upon a second sector, criminal justice, and involves two decision makers.

3.1 Institutional Arrangements, Outcomes of Interest and Opportunity Costs

Within government, responsibility for resource allocation is typically apportioned between departments, each focusing on a particular sector of the economy, and with a budget exogenously allocated by a central decision-making process (e.g. a finance ministry), resulting in a set of distinct agencies differentiated by their remit [18, 19]. The remit of each department is often broad, with multiple objectives, and within each department there may be further apportioning of roles and responsibilities resulting in multiple tiers of principal agent relationships [4]. Typically, departmental decision makers have objectives against which they are judged, some of which may be explicit and clearly defined and others that may be less transparent.

Consideration of the objectives of decision makers can help define the dimensions to be included in the impact inventory. The objectives by their nature should direct the focus towards outcomes of value. Matching the dimensions to the objectives, therefore, aligns the consideration of impacts on individuals with the interests of the relevant decision maker(s).

3.2 The Healthcare Sector and a Single Decision Maker

Healthcare bodies often state the improvement of population health as a key objective [20]. A generic measure of outcome that can be applied across all diseases is preferable for analysis to support resource-allocation decisions. This is because it allows for direct comparison of all direct effects (e.g. health benefits and side effects) with opportunity cost for a given intervention and facilitates consistency in decisions across disease areas. Many different measures of health are possible. The dimensions in the inventory could potentially consist of length of life and a description of the health states experienced using a multi-attribute description system such as the EuroQol 5D questionnaire [21]. However, given the existence of pre-specified generic measures for health, one of the common summary measures that integrates quality and quantity of life lived may be considered an acceptable dimension in its own right, even though it incorporates specific value judgements into the impact inventory. In the UK, for example, the QALY is the preferred generic health outcome [22, 23]Footnote 4 and this has also been reflected in the USA [10].

Other outcomes may be important in healthcare beyond improving health, for example, access to healthcare, patient experience and equity [24]. Where it is not feasible to reflect all dimensions that could be considered important—for example, for reasons of time constraints or availability of evidence—decision makers’ deliberations will have a key role in determining the scope of the impact inventory [6].

To capture the opportunity costs from implementing an intervention requires consideration of what would alternatively have been done with the resources if the intervention had not been funded. Decision makers are not typically tasked with identifying specific interventions that will be forgone and they cannot determine which will be forgone in sectors outside their remit. Instead, the interest is in an estimate of the value of the outcomes from activities that would have been funded in the absence of the specific intervention being considered. This information is potentially knowable, but generating the relevant evidence can be challenging. In healthcare, recent research in the UK estimated the health impacts of changes in spending across the National Health Service (NHS) budget [6, 25]. This provides an empirical estimate of the marginal productivity of the NHS; that is, of the health that will be gained or lost from marginal changes in spending, such as those associated with the introduction of new interventions or policies in the healthcare sector.Footnote 5 An extension to this work has considered which individuals bear the opportunity costs in terms of socioeconomic characteristics and current allocation.Footnote 6 Other countries are also undertaking work to estimate the marginal productivity of their healthcare expenditure [26, 27].

3.2.1 Case study: the example of treatments for individuals with alcohol use disorders

Details of the Second Panel’s case study evaluating treatments for AUDs are shown in Table 1. This shows the published results, detailing the dimensions and costs for two of the strategies (medical management [MM] only and MM + naltrexone). Full details of what is included in each cost dimension are reported elsewhere [10].

Table 1 Cost-effectiveness results for alcohol use disorder treatment
Full size table

To implement the framework, the relevant dimensions and individuals affected need first to be considered, which will depend on which decision maker(s) the analysis is trying to inform. From a healthcare decision-making perspective, we assume the decision maker only cares about population health. Hence, the within-patient aggregation function could simply be the net gain in health, which is the direct effect less the opportunity cost. A within-dimension aggregation function is required to aggregate across patients. The individuals affected would be those who receive the AUD treatment if funded and those who forgo other types of intervention as a result of the resources used to fund AUD treatment not being available for other purposes (i.e. the opportunity costs).

Based on this narrow one-sector analysis, MM + naltrexone is dominant, generating more health (0.1 additional QALYs—see row A) and being less expensive (saving $2193—rows H and I) than MM only, and it appears that MM + naltrexone is worthwhile. If the additional health that can be generated from the cost savings was considered (assuming the resources are spent on healthcareFootnote 7), health gains would be greater as the resources can be used for other patients. Figure 5 sets this out in terms of an impact inventory with the direct effect on health of the AUD patient and the indirect effects on health via opportunity costs on unknown patients. Assuming the $100,000 per QALY threshold used by the Second Panel represents the marginal productivity of the US healthcare sector, the cost savings would be expected to generate an additional 0.022 QALYs.

Fig. 5
figure 5

Impact inventory from a healthcare perspective captured at the level of the average individual with alcohol use disorders and other unknown individuals. AUD alcohol use disorder, Pi individual, QALY quality-adjusted life-year

Full size image

If the decision maker is willing to aggregate across individuals with AUD and the unknown patients, such that the health to each is valued equally, there is a total net health benefit (i.e. health from the intervention being introduced less health if it had not been) of 0.122 QALYs. It should be noted that the widely used $100,000 per QALY threshold for the USA could be seen to represent a societal willingness-to-pay-based estimate rather than an estimate of what health could be produced elsewhere with the same resources based on the system’s marginal productivity. If the latter was lower at, say, $50,000 per QALY, an additional 0.044 QALYs would be generated for other patients [28, 29].

If health is the only dimension of interest for healthcare decision makers, and the aggregation method is acceptable, then an intervention that improves overall health should be approved. Alternatively, the functions could be expanded to consider, for example, characteristics of the individuals whose health is impacted and the initial level of health of each individual, with different weights attached to each individual [30].

3.3 Including Other Dimensions

Health interventions may have impacts on other dimensions that may be considered important [24, 31]. Furthermore, health is not the only dimension that is of potential social value, otherwise all resources would be devoted to the production of health. A broader view of what might constitute social value requires consideration of which dimensions are important.

3.3.1 Impacts Beyond Health But Only One Decision Maker: Health and Consumption

Impacts on individuals’ consumption of other goods are now considered alongside impacts on health.Footnote 8 Consumption here relates to individuals’ purchases of goods in private markets, not their complete consumption of all goods and services. Using the impact inventory, the change in consumption of specific goods could be captured (e.g. the number of apples an individual purchases). However, a composite of change in total consumption using market prices may be acceptable given that individuals determine the purchase of such goods, and it may be reasonable to assume that market prices reflect the marginal value of each good.Footnote 9 The counterpart to consumption is productivity, which refers to the value of goods produced by an individual. If a new intervention results in an individual’s consumption increasing by more than the amount they produce, the excess is supported by the increased production or forgone consumption of other individuals.

The effect of forgone healthcare interventions on consumption and productivity also needs to be considered, and these opportunity costs could be estimated using the marginal productivity of the healthcare sector for consumption and productivity (that is, for all the dimensions of interest, an estimate should be generated of what the opportunity cost will be on that dimension for each sector). A stylized example of the evaluation of a healthcare intervention with impacts on health and consumption is presented in Section A1.2 of the supplementary material.

3.3.2 Application to the Case Study

Figure 6 considers the impacts on the dimensions of health and consumption of the AUD intervention. The impact of healthcare on health remains the same as in Sect. 3.2. There is a gain in individuals’ consumption of $2287 from MM + naltrexone compared with MM only (where consumption includes the effect on the AUD patients’ time valued monetarily and on out-of-pocket costs—rows F, G and K). However, as a result of a smaller increase in the individual’s productivity (only $745—row J), there is a negative net production effect (change in individual’s production less their change in consumption) such that other individuals (Punkown2) would have to forgo $1660 of consumption to fund the AUD patients’ additional consumption. As such, with MM + naltrexone, there is a gain in health and consumption to the AUD patients, a gain in health to other unknown patientsFootnote 10 and a loss of consumption to another group of unknown individuals.

Fig. 6
figure 6

Impact inventory incorporating consumption impact captured at the level of the average individual with alcohol use disorders and other unknown individuals. AUD alcohol use disorder, DE direct effect, OC opportunity cost, Pi individual, QALY quality-adjusted life-year

Full size image

If both health and consumption are important to the decision maker (e.g. for a local area public health decision maker whose mandate extended to include individual consumption), a means of comparison of the health and consumption gained and forgone is required. Either approach to aggregation (within dimension or within individual) requires a normative judgement on how to value the two dimensions relative to one another, i.e. the value of a unit of health relative to consumption (\(v_{\text{h}}\)).Footnote 11 This could be based, for example, on the willingness to pay for health of a sample of the general population, or each individual’s willingness to pay. If we take a within-dimension approach where the decision maker is indifferent to whom the health and consumption accrues, there is a net health gain of 0.122 or 0.144 QALYs (based on a $100,000 or $50,000 per QALY marginal productivity, respectively) and a net consumption gain of $627 (gain to PAUD less forgone consumption for Punknown2 to fund it) generated by MM + naltrexone compared with MM only. If the willingness to trade health for consumption is $100,000 per QALY, there is a total net gain in monetary value of $11,573 (or $13,773). This could alternatively be expressed in terms of health of 0.11573 QALYs (or 0.13773 QALYs).

3.3.3 Extending to Three Dimensions and Two Sectors with Two Separate Decision Makers

Previously, only one sector and one decision maker have been considered. Criminal justice can be considered another sector for which government takes responsibility for the allocation of resources. There are many potential objectives set in the criminal justice system, for example, recidivism rates for probationary services or crime levels for police forces. It could be considered that the ultimate aim is to reduce the level of crime faced by individuals in society. In criminal justice, unlike in health with QALYs, there is currently no established generic measure to capture crime reduction (and all its wider benefits) that would allow consistent comparison across policies. This complicates analyses even when restricted to consideration of criminal justice effects only, let alone when wider impacts are considered. However, for a basic analysis, it may be appropriate to use the number of crimes as the relevant dimension for the criminal justice sector. Opportunity costs will also need to be estimated, based on the costs to each sector and the marginal productivities of each sector for each outcome.

Whether the introduction of the intervention is worthwhile requires consideration of the objectives of the two decision makers involved (i.e. those relating to healthcare and criminal justice). If there are positive net benefits in health, criminal justice and consumption, there would be no conflicts, and decision makers in both sectors would consider the introduction of the intervention to be worthwhile regardless of their weights for the different dimensions. However, in the case of losses in one or two sectors from health, criminal justice or consumption, a method for aggregation is required. As with health and consumption, a means of valuing these on a common metric is required. This could be in terms of the consumption value for the outcome of the criminal justice sector (\(v_{j}\)). If this is not known, then a reasonable initial proxy may be to consider that the allocation of the budgets in society is such that the value of a unit of currency spent in either health or criminal justice is the same in terms of its consumption value.Footnote 12

The use of this framework to make relevant trade-offs explicit does not, of course, guarantee that it will be possible to get consensus between the different decision makers on the method for aggregation and the values used. When decision makers have different objectives or relative valuations of objectives, the net benefit may look different to each. However, by providing the evidence on the impacts on different outcomes, and presenting results based on different relative valuations, the analyst can help to inform any deliberations between decision makers more generally. It also facilitates consideration as to whether it is possible for one decision maker to compensate another so that there are gains in all dimensions [15, 32]. A stylized example of the evaluation of an intervention considering three dimensions (health, education and consumption) and involving two decision makers is presented in Section A1.3 of the supplementary material.

3.3.4 Application to the Case Study

Expanding the previous case study also to consider criminal justice, MM + naltrexone compared with MM only results in fewer crimes and, therefore, fewer years in jail, lower tangible costs of crime, lower criminal justice costs, lower incarceration costs and lower monetarised quality-of-life impacts of crime. A challenge here is that, as presented by the Second Panel, many of these costs do not fall on the same decision makers or individuals. For example, the tangible costs of crime include the costs of healthcare, other public sector activities and property damage, each of which falls on different decision makers and budgets. The Second Panel made a judgement that it is acceptable to aggregate all these into a single dimension (tangible costs of crime). This could raise issues, for example, if the opportunity costs of resources for each budget differs or if different decision makers involved have different views of the value of costs falling on those budgets.

An alternative approach is shown in Fig. 7, which separates the quality-of-life impacts on victims (who are potentially known) from all other ‘legal costs’ (tangible costs of crime, incarceration costs, motor vehicle costsFootnote 13). MM + naltrexone results in direct effects on quality-of-life impacts on victims from crime, which when monetised are equivalent to savings of $691 (row k) to the victims of the AUD patients (Pvictim—with 0.07 crimes averted—row D) and further legal cost savings of $605 (row Q, or sum of rows M, N and O). A ‘criminal justice’ decision maker may be interested in how many crimes are averted. Directly, 0.07 crimes were averted, but if those freed ‘legal’ resources could also be used to avert crimes, some measure of the productivity of those budgets would be required to estimate the total number of crimes averted (e.g. a marginal productivity of the criminal justice system). Alternatively, it may be considered reasonable simply to aggregate the monetarised quality-of-life impact from victims with the other legal costs falling across a range of sectors (the approach taken by the Second Panel, resulting in total legal cost savings of $1296—rows L and Q). Careful consideration now needs to be given to how to aggregate these. One approach is aggregation across outcomes for each individual affected and then across individuals (a within-individual approach). This would require the identification of each individual and the impacts on their health, consumption and crimes. Alternatively, methods for aggregation within dimension could be considered (e.g. how many QALYs are generated, how much additional consumption is generated, etc.). Each approach involves a series of normative judgements.

Fig. 7
figure 7

Impact inventory incorporating criminal justice captured at the level of the average individual with alcohol use disorders and other unknown individuals. AUD alcohol use disorder, DE direct effect, OC opportunity cost, Pi individual, QALY quality-adjusted life-year

Full size image

Finally, it is worth considering the approach taken by the Second Panel. They considered it appropriate to aggregate costs and all dimensions captured monetarily (everything excluding health), to estimate a ‘societal cost’ of the interventions. These were then compared with the health gain based on a ‘cost-effectiveness threshold’ of $100,000 per QALY. Using such an approach, MM + naltrexone results in 0.1 QALY gain and cost savings of $1898, resulting in a total net monetary benefit of $11,898. This approach involves a number of strong value judgements and assumptions. For example, that the opportunity costs falling on all budgets are assumed to be the same, that all other dimensions can appropriately be captured monetarily and that there is indifference with respect to the individuals upon whom the direct effects and opportunity costs fall. These judgements and assumptions may be considered acceptable, but the analyst should make these explicit and identify possible alternatives. The impact inventory proposed here should make these judgements more transparent.

4 Discussion

Economic evaluations are used to inform decisions across different settings that can have very different institutional arrangements. The contribution of this paper is to address collaborative or shared decision making where policies affect resources and outcomes across multiple independent decision makers with different objectives and responsibilities. This framework describes both how economic evaluation can inform these decision makers and how to conceptualise a societal perspective by identifying which dimensions require trade-offs to resolve differences between decision makers. It seeks to remain neutral with respect to the normative considerations inherent in all forms of economic evaluation and the form of economic evaluation, aiming to distinguish the points at which value judgements feed into the evaluation process and the implications of alternative judgements where possible.Footnote 14

The framework proposed here can be seen as a broader, extended version of the ‘impact inventory’ suggested by the Second Panel [10]. It obliges the inclusion of opportunity costs, which are not explicit in the Panel’s approach. The approach here makes explicit relevant normative judgements, whereas the Second Panel arguably imposes a specific aggregation function within dimension, whereby all individuals are valued equally and across dimensions and where all non-health dimensions (e.g. quality-of-life impacts of crime) can be captured monetarily and aggregated with costs.

The range of alternative normative principles that can help to define social value have been extensively researched and are inevitably contested. Two broad normative frameworks that underlie the economic evaluation of healthcare interventions are welfarism and extra-welfarism [33]. Welfarism states that social welfare is a function of individual utility and, therefore, would require a within-individual evaluation within the framework presented here. Extra welfarism is compatible with both within-individual and within-dimension approaches. Either way, to define social value using an explicit social welfare function defined across individuals and dimensions requires that the full set of dimensions and the methods for aggregation be defined ex ante. For this to be useful for decision makers, each would have to agree that the function is appropriate and that they will follow its implications for policy. This is likely to be challenging given the many conflicting and contradictory claims on what is socially valuable.

The framework can be used across different forms of economic evaluation such as cost-effectiveness analysis or cost–benefit analysis. Other approaches proposed for the evaluation of policies with wide impacts include social rate of return [34], universal outcome measures [35] and multicriteria decision analysis [36], but there is generally a particular set of value judgements implicit in each. For example, the social rate of return analysis aggregates across all dimensions using monetary values, the source of which may be contestable, and often ignores opportunity costs in different sectors. The universal outcome measure approach uses an outcome taken as relevant across all sectors (e.g. well-being) and assumes the underlying dimensions and their relative values are known and accepted by all.

The extended impact inventory presented here emphasises the importance of a disaggregated presentation of costs, effects and opportunity costs by dimension. The framework shows the changes in relevant dimensions from an intervention, and how the subsequent application of values establishes whether the intervention is worthwhile. Whose values should be used for this purpose is, of course, a political question. Some may wish to be prescriptive about this by, for example, specifying the rate at which health should be traded for education, thus defining a (partial) social value function. The basis of the applied examples presented in this paper can be considered consistent with the ‘social decision-making approach’ which aims to consider how society establishes processes to balance conflicting and contradictory claims on what is valuable [5, 37]. The implications of this process in terms, for example, of the current trade-offs between health and education and the budgets made available to decision makers, can be seen as providing a partial but legitimate expression of some unknown underlying social value function [38]. Even if it is felt that the current budgetary allocation does not accurately reflect societal values, it determines the opportunity costs.

Given the objectives and responsibilities of different institutions, it can be regarded as acceptable that public sector decision makers determine the values to apply or their source (e.g. the preferences of a sample of the public). Some decision-making organisations have publicly defined their preferred approaches with ‘methods guidelines’ (e.g. drug reimbursement authorities internationally). Where this is not the case, it may be helpful to start with a ‘base-case’ set of value judgements that reflect those used in similar exercises for the relevant organisations, and this reflects the approach taken in the case study here. Importantly, however, the value judgements in the base case need to be explicit, alternatives made available and their importance to overall conclusions made clear. By providing the evidence on the impacts on different dimensions, and presenting results based on a range of valuations, the analyst can help to inform deliberations between decision makers responsible for different sectors. This contrasts with the implicit value judgements taken in many economic evaluations, particularly those claiming to be taking a ‘societal perspective’.

Furthermore, by making decision makers’ value judgements explicit, future evaluations for the same decision makers can apply those as one approach to valuation and aggregation while assessing the impact of others. This has implications for the practicality of using the framework, making it consistent and easier to implement over time. Even where a particular decision maker has no interest in the consequences of their choices on other dimensions, by presenting the full range of evidence across all affected parties the analyst has shown the trade-offs and, therefore, the implications of the decision maker’s limited view. A deliberative decision-making process can, therefore, be informed, and potentially held accountable by an explicitly partial analysis capturing the major dimensions impacted upon. The results from this partial approach may also be valuable to inform budget reallocation negotiations by highlighting any discrepancy in marginal productivity across sectors and the budget transfers that this implies.

Achieving consensus in the dimensions considered socially valuable, and their relative values, may be an impossible task. However, decisions still need to be made, and by offering assessments of the impacts (the direct benefits and opportunity costs) on those dimensions that are considered most important, the analyst can help to inform these decisions through quantitative analysis. This approach also allows for the consideration of potential transfers between decision makers to compensate winners and losers. These analyses provide a strong basis for assessing the value of new policies.