Advertisement

EURO Journal on Decision Processes

, Volume 4, Issue 3–4, pp 161–181 | Cite as

Using MCDA to generate and interpret evidence to inform local government investment in public health

  • Brian P. Reddy
  • Praveen Thokala
  • Alison Iliff
  • Kerry Warhurst
  • Helen Chambers
  • Lynsey Bowker
  • Stephen J. Walters
  • Alejandra Duenas
  • Michael P. Kelly
Open Access
Original Article

Abstract

Smoking is the single biggest cause of preventable death in the Uited Kingdom (UK) and is a major cause of coronary heart disease, some cancers, and respiratory disease, including chronic obstructive pulmonary disease. At the time of initiating the project, smoking prevalence had not changed across four local government areas in South Yorkshire for some years. Most spending had been focussed on helping people quit, an intervention where there was clear evidence of effectiveness. A number of changes occurred in public health structures and targets, requiring a reappraisal of the range of interventions offered. This was challenging due to a lack of clear evidence for some of the areas’ alternative interventions. The aim of this paper is to describe the use of a multi-criteria decision analysis (MCDA) approach to support the health priority setting in local authorities to reduce smoking prevalence. There were three phases to this process: (1) problem structuring; (2) the multiple criteria decision analysis; (3) and using the MCDA results to influence decision making at the local government level. The MCDA approach was used to collate information in a consistent and transparent manner, using expert, stakeholder and public opinion to fill known gaps in evidence. Fifteen interventions (such as stop smoking support services, smoke-free spaces, communication and marketing exercises, and increased investment in enforcement) were ranked across eight criteria (relating to reductions in prevalence across relevant groups, as well as aspects relating to equity and feasibility), allowing a range of relevant concerns to be incorporated. Subsequent steps were taken to translate the results of this stage into workable policy options. The results differed significantly from current practice. Sensitivity analysis showed that the findings were robust to changes in preference weights. These results informed subsequent changes to the interventions offered across the four boroughs. The ability of MCDA techniques to incorporate data and both qualitative and quantitative judgements in a formal manner mean that they are well suited to support public health decision making, where evidence is often only partially available and many policies are value driven. MCDA methods, if used, should be chosen carefully based on their resource/time constraints, scientific validity, and the significance and broader context of the decision problem.

Keywords

MCDA Public health policy Health economics Tobacco interventions Evidence-based decision making 

1 Background

Smoking is the single biggest cause of preventable death in the United Kingdom (UK). Smoking is a major cause of coronary heart disease, some cancers and respiratory disease, including chronic obstructive pulmonary disease. These diseases have the greatest impact on the least well off in society and it is estimated that up to half the difference in premature death between the most and least affluent groups is explained by different patterns in smoking (Jarvis and Wardle 1999). Despite high tax rates on tobacco products, the total societal cost of smoking, in the UK, was estimated in 2010 at £13.74 billion (Nash and Featherstone 2010), representing a net loss to society of 6.5 p per cigarette smoked. In the “Healthy Lives, Healthy People: a Tobacco Control Plan for England” white paper (UK Government 2010), the UK government set out the ‘national ambitions’ in reducing smoking prevalence for the coming 5 years:
  • to reduce adult (aged 18 or over) smoking prevalence in England to 18.5 % or less by the end of 2015 (from 21.2 %), meaning around 210,000 fewer smokers per year;

  • to reduce the rates of regular smoking among 15-year-olds in England to 12 % or less (from 15 %) by the end of 2015;

  • to reduce the rates of smoking throughout pregnancy to 11 % or less (from 14 %) by the end of 2015 (as measured at the time if giving birth).

The metropolitan county of South Yorkshire is made up of four local authorities (Barnsley, Doncaster, Rotherham and Sheffield) and is home to approximately 1.3 million people. Each local authority acts independently and has its own director of public health (DPH). Each local authority has similar-sized budgets relating to tobacco and a tobacco control commissioner (TCC) responsible for the area. The overall smoking prevalence rates in the local authorities had not changed for a number of years, and these rates were higher at the time the project was initiated than the national average for England of 19.5 %: from 22.7 % in Rotherham to 25.3 % in Doncaster (Office for National Statistics 2012). This status quo was no longer felt to be tenable and a new approach was deemed necessary by the TCCs, who approached the School of Health and Related Research (ScHARR) at the University of Sheffield, hoping to identify an appropriate ‘knowledge-based approach’ that might inform future local strategies to reduce smoking prevalence in a rational and transparent way.

This paper aims to describe the use of a multi-criteria decision analysis (MCDA) to support health priority setting in local authorities to reduce smoking prevalence. A multi-criteria decision analysis approach was proposed and used, incorporating the views of the TCCs, the public, and an assembled expert and stakeholder group, to gauge a range of opinion, to present a range of policy options to each borough’s respective DPH. The next section describes the literature review conducted to identify approaches for local-level priority setting. Section 3 describes the MCDA methods used in our study and the discussion (Sect. 4) highlights the relative advantages or disadvantages of our approach. Section 5 presents the conclusions.

2 Literature review on priority setting at a local level

This section provides an overview of methods used to support health priority setting in local authorities. Some of these methods have been developed in health care independent of MCDA, but share a number of similarities. The priority setting approaches that drawn on MCDA techniques, including the Portsmouth Scorecard, Option Appraisal, Socio-Technical Allocation of Resources (Star), as well as Programme Budgeting and Marginal Analysis (PBMA).

In the Portsmouth Scorecard, each option is scored against selected criteria and total scores are calculated and discussed. A single index score is calculated for each intervention with the highest scoring interventions taking priority (Williams et al. 2011). This approach is easy to understand by decision makers and the criteria can be adapted to suit the priority setting context (Robinson et al. 2012). Different versions of the Portsmouth Scorecard have been developed to inform decisions over clinical and non-clinical interventions in a number of localities across England (Robinson et al. 2012).

Option Appraisal (Brazier 1987; Akehurst et al. 1988), another simple decision analysis technique, has been in use since the 1980s. It involves defining objectives, examining options and weighing up the costs, benefits, risks and uncertainties (HM Treasury 2003; UK Government 2011). Examples of Option Appraisals include mental health services in Ashton, Leigh and Wigan (2005) a well as Aberdeen City Council (2010) and coronary heart disease monitoring in Scotland (Perry et al. 2000).

Airoldi et al. (2014) used MCDA techniques to structure a prioritisation process at the Isle of Wight Primary Care Trust (PCT), which has since been developed into the Socio-Technical Allocation of Resources (Star) toolkit in collaboration with the Health Foundation. This method comprises decision conferencing for stakeholder engagement and a value for money analysis. An Excel-based tool is used alongside a facilitated stakeholder workshop and the results are presented using visual aids such as value-for-money triangles and efficiency frontiers. Examples of Star include the Isle of Wight (Airoldi et al. 2014) and Sheffield PCT (Williams et al. 2011; Airoldi et al. 2014; Airoldi 2013; Health Foundation 2012).

Programme Budgeting and Marginal Analysis (PBMA) consists of identifying resources and how they are being spent in specified programmes (PB) before considering the expected incremental costs and effects of the proposed investments/disinvestments at the margin (MA), to maximise benefit and minimise costs (Wilson and Scott 1995; Donaldson 1995; Madden et al. 1995). A broad framework of PBMA is presented by Mitton and Donaldson (Mitton and Donaldson 2003), which has been applied for example in gynaecological services (Twaddle and Walker 1995), diabetes care in general practices (Scott et al. 1998), respiratory disease (Bohmer et al. 2001) and coronary heart disease (Haas et al. 2001).

MCDA approaches have been used in a wide range of public health settings at non-local authority level such as investigating government drug policy (Nutt et al. 2010), shared decision making to help choose the most appropriate screening procedure for a patient (Dolan 2000) and estimating the risk from food-borne diseases (Ruzante et al. 2010). They have also been widely used in non-health settings, especially those related to environmental contexts. The United Nations has used it in a number of areas, such as climate change (United Nations Framework for Climate Change 2002) and fisheries management (Soma 2003); others have used it for areas such as water safety (Lindhe et al. 2013), forestry (Sheppard and Meitner 2005) and waste management (Hanan et al. 2013). It has also been used in UK public sector settings outside of health to provide yes/no decisions, such as the UK Border Agency’s point-based system on whether to allow citizens from outside the European Economic Area to work and settle in the UK (UK Border Agency 2010). However, a key aim of this study was to design an appropriate model for use at a local government level, given the specific problem at hand.

There are a number of different MCDA methods for priority setting. Issues of availability of data, the very purpose of the approach, its next steps, and availability of time to carry out the process are important in making final decisions regarding which MCDA approach to use and which compromises might be necessary and acceptable. The sophistication of the method can range from simple methods (e.g., Portsmouth Scorecard) to more complex methods (e.g., PBMA) as shown in Fig. 1. Another aspect of consideration for priority setting is the availability of evidence. As seen in Fig. 1, data on the alternatives’ performance on each of the criteria can be gathered in a variety of ways, ranging from evidence synthesis to expert opinions.
Fig. 1

Field showing range of possible MCDA-based approaches, described by evidence requirements and resource/time use

Whilst these applications in Fig. 1 are all based on MCDA principles, they vary significantly in terms of scientific rigour, robustness and their requirements regarding participants’ time and other resources. The MCDA method used in our study, which we have called the South Yorkshire Multi-criteria Decision Analysis Prioritisation for Local Effectiveness (SYMPLE) was chosen based on the priorities and constraints relating to resources and time, as well as scientific validity and the significance and broader context of the decision problem.

3 Methods

This section describes the details of the MCDA approach used in our study. There were three distinct phases to (which are split into 12 stages) in our MCDA approach:
  1. 1.

    Problem structuring.

     
  2. 2.

    MCDA modelling.

     
  3. 3.

    Supporting decision making.

     
The 12 stages are listed in Table 1, with the three phases above separated by dashed lines. The sequence of stages allowed the views of the general public, expert opinion and other stakeholders to be incorporated at various stages. The findings were ultimately used as a basis for prioritisation among the options under consideration. The stages were meant as a rough guide and revisited as necessary throughout the process—the entire process was therefore best seen as a framework for investigating and discussing the objectives and issues rather than a rigid formula.
Table 1

Involvement by relevant groups in project’s stages and phases

 

Project phases and stages

General public

Stakeholder expert group

Tobacco control commissioners

Lead researcher

Local decision makers

Problem structuring

Establish the decision context

  

 

Identify relevant stakeholder expert steering group

  

  

Identify intervention alternatives

  

  

Decide on appropriate MCDA technique to combine criteria

   

 

MCDA modelling

Identify relevant criteria to the decision problem

 

‘Weighting’, reflecting relative importance of each criterion to the decision

  

Estimate performance of the interventions on the criteria by gathering evidence or expert opinion

 

   

Estimate the overall score of each intervention, which can be used for prioritisation

  

Supporting decision making

Examine the results

  

 

Sensitivity analyses

   

 

Implementing findings

  

 

3.1 Problem structuring

This stage required explicit articulation the aims of the process, the list of stakeholders potentially impacted and who should be responsible for making the final decision. This needed to be designed cognisant of the broader decision-making context and the potential consequences of using such a novel technique in this setting, given that there would be inevitable winners and losers from any decision. Consideration of both scientific/technical and social/human factors using a ‘socio-technical’ approach (Dodgson et al. 2009) is something of an art, but fundamental to the success of such a process.

3.1.1 Stakeholders

The TCCs identified an expert steering group made up of a wide variety of stakeholders to support the prioritisation process. These stakeholders came from a range of relevant settings with a role to play in reducing tobacco consumption, including specialities such as current stop smoking providers, midwifery, elected councillors, the fire service and a local chamber of commerce. This allowed for a wide range of views to be included and potentially increases the legitimacy of the findings, as practically all relevant parties were represented in the process.

3.1.2 Alternatives

Fifteen interventions were examined, chosen to represent a broad range of potential techniques. Five of the six strands advocated by the World Health Organisation (2003) and the UK government (All-Party Parliamentary Group on Smoking and Health 2010) to overcome practical barriers in tobacco control were included; the only such theme not represented was reducing tobacco advertising and promotion, which was felt to be outside the scope of local government control. Stop smoking support (SSS) services—which previously constituted around 90 % of tobacco spending in each local authority—were investigated alongside other interventions to ensure increased clarity and in the hope that a more effective holistic approach might emerge. There are potentially a limitless number of interventions available at various levels of investment, but MCDA techniques could not investigate every such intervention in practice. Given the findings would still be subject to political factors, the list was intended to be broad and representative rather than comprehensive. Three- to four-page business cases for each intervention were prepared, giving a broad outline of how the intervention would likely be implemented in practice and written in language accessible to all stakeholders. Brief descriptions of the interventions are available in the appendix. The interventions considered (grouped by the WHO ‘strands’) were:
  • targeted stop smoking support (SSS) services:
    • for routine and manual socioeconomic groups (R&M);

    • for people with long-term conditions;

    • for parents;

    • for black and minority ethnic communities (BME);

    • for people with mental health conditions;

    • pre-operatively;

    • in secondary care in hospital;

    • in maternity settings;

  • reducing exposure to secondhand smoke;
    • extend smoke-free spaces beyond legislation by voluntary extensions into areas families use;

    • SoNIC (“Social Norms in Local Communities”) programme to influence behaviour change;

  • media and education campaigns;
    • mass media, population-based communications and marketing campaigns;

    • customised, ongoing children and young people education programme;

    • replication of “A Stop Smoking in School Trial” (ASSIST) programme;

    • training for public sector staff, businesses and communities to provide brief advice to smokers and those affected by smoking;

  • tobacco regulation/reducing availability of tobacco;
    • enforcement to enhance the statutory functions of the local authority to enforce legislation on under-age sales and the selling of illicit tobacco.

3.1.3 Choice of the MCDA method

A standard linear additive MCDA model was chosen as the most appropriate approach for this decision problem. This is the most commonly used approach in health care (Marsh et al. 2014). Swing weighting was used to weight the criteria as it allows for the trade-offs which are inevitable in public health decision prioritisation (Baltussen and Niessen 2006) and because such weighting fulfils the theoretical requirements of MCDA (Belton and Stewart 2002). Scoring was performed using direct rating as they were elicited from the expert group. An emphasis was placed on achieving consensus by bringing stakeholders together at a meeting to debate and score topics, thus encouraging their understanding of the final decision and increasing their ‘buy-in’ (Phillips 2007). As a result, the ‘social’ aspects of the socio-technical process at the meeting had to be carefully managed (and are further described in Sect. 3.2.3 below). Approaches that are typically carried out by individuals, such as DCE—rather than those that required discussion and cooperation—were not considered appropriate on this basis. Only one-half day was available to bring together the expert group together. Thus, we chose a simple approach. Previous experience had suggested that the analytic hierarchy process may be too slow moving for such a group (Reddy et al. 2014) and was therefore not used. The total scores were calculated for each intervention, using a simple weighted sum method, which was then used to prioritise among the interventions.

3.2 MCDA modelling

To create a complete ranking, stakeholders had to specify the decision’s relevant criteria (and their relative importance) and the performance of the alternative interventions on each.

3.2.1 Criteria

The draft list of suitable criteria was identified through public consultation. Workshops were set up to engage with volunteers from the public to discuss a series of potential interventions and describe why they would prefer investment in one rather than another. These criteria were ranked according to participants’ views. They were then sent out to the expert group for comment and ranking and were subsequently revised. These steps were non-binding, but were used to inform the subsequent debate on which criteria to use.

A number of iterations were required to arrive at the final workable set of criteria. Some criteria likely to be highly related had to be removed or revised to avoid double counting, which might have biased the results. Others, such as feasibility, were removed as all alternatives under consideration were a priori felt to be similarly implementable. This process took some time, but once consensus was reached it was felt to have been an extremely valuable exercise in building a shared understanding of what makes an intervention worthwhile and in identifying underlying goals of the tobacco programme across all the strands. Overall six headline criteria were identified, as shown in Table 2. Given difficulties in generating (and subsequently communicating) criteria and their weights by correspondence, these were by necessity defined quite broadly, as themes to be considered as part of the discussions on the day of the workshop.
Table 2

Inputs to MCDA process prior to decision conference: the criteria, and their weightings

Criterion definition

Considerations

Weight (%)

How well does it meet our priorities on the national indicators to:

Reduce smoking amongst adults

Reduce smoking amongst young people

Reduce smoking amongst pregnant women

50

(20)

(15)

(15)

How much will this improve a person’s health or quality of life?

Will it improve their heath or help them to live longer?

Are there other aspects of a person’s life that will improve; money, better environment, lower crime etc?

15

Will it help the difference in health or length of life between our communities

Can the people who need this intervention get to it?

Will it reach our high risk groups and communities?

Will people use the service or intervention?

15

Value for money

Does it work?

How many people will benefit - both directly and indirectly?

Can we afford to do it?

10

Ease of implementation

Will it contribute to and enhance services/pathways?

Can we get the right staff, buildings, equipment etc.?

5

Sustainability

Is it sustainable?

Are there any risks associated with this intervention?

5

3.2.2 Weighting

The preference weights for the criteria elicited using swing weighting are also shown in Table 2. These synthesised the viewpoints of the expert panel, the TCCs and the level of difference between the best and worst performing intervention on each criterion. Many of the expert panel expressed a view that large impacts to individuals should be preferred to interventions that have small impacts to larger numbers, and that equity aspects should be considered from the beginning (rather than after cost-effectiveness analyses had been carried out). In contrast, the main concern of the TCCs was the prevalence rates described in the national priorities. The importance of each criterion was elicited by correspondence before the meeting of the expert group due to expected time constraints on the day. These were converted into swing weights by the lead researcher based on deliberation with the TCCs regarding the range from best- to worst-performing intervention on each criterion. While not ideal, this was felt to be a necessary compromise. Sensitivity analyses were later used to ensure that the results were meaningful, in light of the fact that the expert group was not always in complete agreement on weightings; these weightings were found to be robust.

3.2.3 Scoring

The next stage required scores to be attributed to each intervention on each criterion. Due to the scarcity of evidence of the interventions on the criteria, it was felt that the subjective opinion of the key experts should be used to elicit the scores. This was carried out in a ‘decision conference’ setting (Phillips and Bana e Costa 2007). To do so, an impartial facilitator worked iteratively with stakeholders to generate an explicit model intended to help those present to think more clearly about the relevant issues. Care was taken to ensure that as broad a range of potential stakeholders were included in the process as possible. About 30 participants from the expert stakeholder group took part, representing the range of specialities from inside and outside health care, and chosen from across the four boroughs.

Participants were initially split into four groups, to ensure that all participants would have the opportunity to voice their opinion at each stage. Participants were assigned according to speciality and borough to ensure a similar mix on each table. They were asked to consider how an intervention was likely to perform on each criterion, in batches of four interventions at a time, and agree upon a consensus score from 0 to 10 based on these discussions, where 0 indicates having the least imaginable impact for that criterion and 10 indicated the biggest imaginable impact (and other intermediate descriptions provided). Tables then fed back their scores to the floor, and debate could occur between groups to ensure an overall consensus score for each intervention on each criterion that emerged. Due to time pressure towards the end of the day, two larger groups were formed to score the final interventions in parallel. The final scores elicited for the interventions are presented in Table 3.
Table 3

Consensus scores generated at decision conference for each intervention on each criterion

 

Reduce smoking amongst adults

Reduce smoking amongst young people

Reduce smoking amongst pregnant women

How much will this improve a person’s health or quality of life?

Will it help the difference in health between our communities?

Value for money

Ease of implementation

Sustainability

Total score

Weighting

20

15

15

15

15

10

5

5

100

Enforcement

7

8

7

7

6

7

8

8

71.0

Maternity stop smoking support

3

5

8

8

7

9

9

9

66.0

Communications and marketing

8

6

7

5

5

8

7

4

64.0

Training for brief interventions

7

5

5

5

7

7

4

7

59.5

Social norms to influence behaviour change

6

8

5

6

6

5

4

5

59.0

Young people education programme

5

6

4

7

6

5

5

6

55.0

Routine/manual stop smoking support

5

4

4

6

6

6

8

8

54.0

Long term conditions stop smoking support

5

3

3

7

6

7

8

7

53.0

Parents stop smoking support

4

6

6

6

4

5

5

6

51.5

Black/minority ethnic stop smoking support—groups who thought it offered high “Value for money” (VfM)

4

3

2

4

6

8

6

5

44.0

Smoke-free spaces—only open spaces

3

4

4

3

3

7

7

7

41.0

Black/minority ethnic stop smoking support—low VfM tables

3

4

4

3

3

3

7

7

37.0

Smoke-free spaces—private spaces

3

4

4

3

3

3

2

7

34.5

Mental health stop smoking support—high VfM groups

2

2

2

5

4

6

5

5

34.5

Pre-operative stop smoking support

2

1

1

7

3

8

4

5

34.5

Hospital stop smoking support

3

1

1

5

3

5

2

4

29.0

Mental health stop smoking support—low VfM tables

2

1

1

7

3

2

4

5

28.5

Replication of A Stop Smoking in School Trial (ASSIST) programme

2

4

1

2

2

2

1

1

20.5

Consensus could not be reached on all intervention regarding their value for money, so these are presented separately, and differences underlined

3.2.4 Aggregation

Scores elicited from participants on each intervention were combined with the weight on each criterion to calculate the total scores for each intervention under consideration. This was performed using a weighted sum approach as shown below:
$$V_{\text{j}} = \sum\limits_{0}^{i} {s_{ij} \times w_{i} } ,$$
where V j is the overall value for intervention j estimated from MCDA model, s ij is the score for intervention j on criterion i and w i is the weight attached to criterion i.

It is important to reiterate that the primary result of the model should be to aid the process of structuring and understanding the decision, rather than necessarily providing some ‘correct ranking’ (Phillips 2007). MCDA practitioners often speak of the importance of parsimony or ‘requisiteness’ in such circumstances (Phillips 1984). The ultimate responsibility to consider all relevant factors (including those not contained within in the model) rested with the relevant directors of public health. Nonetheless, the interventions could produce initial rankings based on their total scores, as shown in Table 3.

3.3 Supporting decision making

Before making their final decisions, the directors of public health (as advised by the TCCs) needed to be confident in the results of the approach and to better understand the robustness of the findings. The following stages investigated the applicability of the results for use in practice.

3.3.1 Examining the results

The results implied that increased coordination with local authority trading standards teams, as advocated in the “Enforcement” intervention, best meets the range of issues relevant to the concerned local authorities. Maternity stop smoking services and investment in communications and marketing, amongst others, also performed well. A number of approaches could potentially be used together in the future. A school-based programme for teenagers, entitled ASSIST, performed very poorly as did Stop Smoking Services relating to hospital inpatients and mental health. Differences in interpretation of some interventions (some participants at the meeting felt it would be infeasible or counter-productive to attempt to ban smoking in parks for example) meant some alternatives were given two sets of scores. Scores were reported separately for the different groups based on whether they thought the interventions offered high value for money (VfM) or low VfM, though this did not change the results notably. Table 3 lists these differing potential interventions independently of each other, to allow comparison.

3.3.2 Sensitivity analysis

The MCDA process is only a decision support tool and it would be inappropriate to take the rankings as a ‘correct answer’ without further examination. We performed sensitivity analyses to investigate the robustness of the findings, which required investigating by how much the weighting on each criterion would have to change to change the final ordering of interventions. These are difficult to illustrate here across 15 alternatives on 8 criteria, but, generally speaking, the magnitude of change in weighting required to alter rankings implies the results are stable. The only exceptions were for those interventions with equal total scores [smoke-free spaces (private spaces only); mental health stop smoking support (high value for money); and preoperative stop smoking support services] for whom any changes in weighting naturally changed their ordering. These three interventions performed relatively poorly anyway and were unlikely to be funded. The smallest change to alter the rankings was if the weighting on reducing prevalence amongst adults was reduced from 20 % of weighting to 18.75 %, maternity SSS overtook it as leader. Because these were the two highest ranked criteria, both were likely to be provided regardless, meaning in practice that the model is in practice more robust than even this. The robustness of the findings is typical in MCDA techniques (Von Winterfeldt and Edwards 1986) and also explains why for different interpretations from the panel (and hence different scores) for certain criteria, the scores attributed had little overall impact on rankings.

3.3.3 Implementing findings

The results firmly show that enforcement, maternity SSS and preventative measures (except ASSIST) appear to perform well in all circumstances according to these criteria. The four councils have reduced the overall spending on SSS services, and realigned spending to focus more on pregnant women in particular. Spending has increased across the board on enforcement, with at least two councils also hiring enforcement officers—which is all the more impressive given the widespread staff cuts. Barnsley also hired a stop smoking midwife on the strength of the report. Education campaigns for young people have been prioritised as a preventative measure. Sheffield has also increased spending on smoke-free spaces, and at the time was the only public health topic to have its budget protected.

4 Discussion

The paper has described and demonstrated that an MCDA approach can be used to support health priority-setting in local authorities to reduce smoking prevalence. The MCDA approach used an open, deliberative process, bringing together experts and stakeholders to rank interventions and making explicit any assumptions. This was the vehicle by which an initial impasse over appropriate areas for outcomes of tobacco spending was transformed into tangible policy options, addressing a range of relevant issues. The open and transparent nature of the approach allowed public and expert opinion to be used at various stages of the process as required, increasing the likelihood of political acceptability and potentially increasing the fairness of final decisions (Daniels 2000). The approach achieved more than just scoring potential actions; it empowered all four councils to make significant changes in the services commissioned and served as something of a uniting mission statement for the project.

It was felt by the TCCs a priori that the results must show legitimacy to be implementable and hence a broad range of stakeholders and public viewpoints would need to be included as part of the process. This ensured that a variety of perspectives are accounted for and that the stakeholders’ (including both ‘winners’ and ‘losers’) own buy into the process, which is important in ensuring successful implementation (Phillips 1984). The large number of stakeholders involved created challenges in terms of communication, which made the process more disjointed than would be ideal. The generation of criteria, weighting and scoring therefore had to be carried out separately and subsequently sewn together by the lead researcher and TCCs. It also impacted on the length of time available at the decision conference, influencing the choice of the MCDA approach used. Criteria were also by necessity defined broadly, though at the meeting the expert group (who had played a major role in devising them) seemed broadly in agreement as to their purpose and relative importance. Sensitivity analyses found these to be robust, and this stage is vital in such circumstances where there is the risk of confusion or disagreement.

Participants and DsPH seemed happy with the results; however, there were some counter-intuitive findings. Most notably, ASSIST came last on the list, despite it being one of the few available interventions for prevention of smoking supported by sufficient evidence of effectiveness that it is recommended by the National Institute for Health and Care Excellence (NICE). It was felt by the expert stakeholders to be unsuitable, given the local environment (especially, due to the perceived risk of schoolyard bullying). As the local government has assumed extra powers since the passage of the Health and Social Care Act, 2013, this may lead to similar conflicts in future between national and local-level decision makers, both in terms of relevant evidence and local preferences.

The results of this approach were only one stage as part of a political process and would only inform the ultimate decisions of the respective directors of public health. The model’s recommendations therefore had to jump another hurdle to ensure political feasibility and common sense. The DsPH could also consider whether there were synergies among interventions (and indeed if further savings were possible by collaborating with other boroughs), neither of which was considered in our process.

Each stage of our study was designed pragmatically to ensure that it had the legitimacy of a wide range of stakeholders and could give a good overview of the likely impact of the interventions under consideration, while taking into account a range of relevant constraints in doing so. Though the approach chosen cannot necessarily therefore be said to conform to ideal best practice, such compromises are often necessary in real-world settings. The programme emerged from a situation where clear evidence of effectiveness was impossible and circumstances dictated that insights would have to be drawn from expert evidence. We attempted to source this in an as systematic a manner as was possible in the circumstances given the context. Ensuring that such context is taken into account was fundamental in ensuring the success of the project.

5 Conclusions

An MCDA approach was used to reappraise public health spending on reducing the harm caused by tobacco across the four local authorities making up South Yorkshire. The ability of MCDA techniques to incorporate data as well as both qualitative and quantitative judgements in a formal manner means that they are well suited to support public health decision making, in which evidence is often only partially available and many policies are value driven. A number of MCDA methods exist for local resource allocation, ranging from quick and simple methods such as the Portsmouth Scorecard to more sophisticated methods such as PBMA. The local decision makers need to choose the appropriate method based on their resource/time constraints, scientific validity, and the significance and broader context of the decision problem.

Notes

Acknowledgments

We would like to thank the participants, who gave their time voluntarily and without which this work would not have been possible. The work was supported by ESRC CASE award ES/I026088/1, entitled “Multi-criteria approaches to public health decision making”, with NICE as the non-academic partner. The ESRC had no role in preparing this document. Full ethics approval for the project was given in advance by the School of Health and Related Research (ScHARR) ethics committee in the University of Sheffield. The authors have no competing interests.

References

  1. Aberdeen City CHP (2010) Final option appraisal for health service redesign for children with complex needsGoogle Scholar
  2. Airoldi M (2013) Disinvestments in practice: overcoming resistance to change through a sociotechnical approach with local stakeholders. J Health Polit Policy Law 38(6):1149–1171CrossRefGoogle Scholar
  3. Airoldi M et al (2014) STAR—people-powered prioritization a 21st-century solution to allocation headaches. Med Decis Making 34(8):965–975CrossRefGoogle Scholar
  4. Akehurst R, Brazier JE, Jeavans R (1988) Prioritising schemes for inclusion in Yorkshire Capital Programme. UK: Health Economics ConsortiumGoogle Scholar
  5. All-Party Parliamentary Group on Smoking and Health (2010) Inquiry into the effectiveness and cost-effectiveness of tobacco control: submission to the 2010 spending review and public health white paper consultation process. All Party Parliamentary Group on Smoking and Health, UKGoogle Scholar
  6. Ashton LaWPCT (2005) Options appraisal for the management, organisation and delivery of mental health servicesGoogle Scholar
  7. Baltussen R, Niessen L (2006) Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Eff Resour Alloc 4(1):14CrossRefGoogle Scholar
  8. Belton V, Stewart T (2002) Multiple criteria decision analysis: an integrated approach. Kluwer, BostonCrossRefGoogle Scholar
  9. Bohmer P et al (2001) Maximising health gain within available resources in the New Zealand public health system. Health Policy 55(1):37–50CrossRefGoogle Scholar
  10. Brazier J (1987) Study of Accident and Emergency Services in East Yorkshire Health Authority. Health Economics Consortium, UKGoogle Scholar
  11. Daniels N (2000) Accountability for reasonableness: establishing a fair process for priority setting is easier than agreeing on principles. BMJ Br Med J 321(7272):1300CrossRefGoogle Scholar
  12. Dodgson J et al (2009) Multi-criteria analysis: a manual. Department for Communities and Local Government, LondonGoogle Scholar
  13. Dolan JG (2000) Involving patients in decisions regarding preventive health interventions using the analytic hierarchy process. Health Expect 3(1):37–45CrossRefGoogle Scholar
  14. Donaldson C (1995) Economics, public health and health care purchasing: reinventing the wheel? Health Policy 33(2):79–90CrossRefGoogle Scholar
  15. Haas M et al (2001) Using programme budgeting and marginal analysis to assist population based strategic planning for coronary heart disease. Health Policy 55(3):173–186CrossRefGoogle Scholar
  16. Hanan D, Burnley S, Cooke D (2013) A multi-criteria decision analysis assessment of waste paper management options. Waste Manag 33(3):566–573CrossRefGoogle Scholar
  17. Health Foundation (2012) Looking for value in hard timesGoogle Scholar
  18. HM Treasury (2003) The Green Book: appraisal and evaluation in central government: treasury guidance. Stationery Office, LondonGoogle Scholar
  19. Jarvis MJ, Wardle J (1999) Social patterning of individual health behaviours: the case of cigarette smoking, vol 90. Social determinants of health. Oxford University Press, OxfordGoogle Scholar
  20. Lindhe A et al (2013) Uncertainty modelling in multi-criteria analysis of water safety measures. Environ Syst Decis 33(2):195–208CrossRefGoogle Scholar
  21. Madden L et al (1995) Public health and economics in tandem: programme budgeting, marginal analysis and priority setting in practice. Health Policy 33(2):161–168CrossRefGoogle Scholar
  22. Marsh K et al (2014) Assessing the value of healthcare interventions using multi-criteria decision analysis: a review of the literature. Pharmacoeconomics 32(4):345–365CrossRefGoogle Scholar
  23. Mitton CR, Donaldson C (2003) Setting priorities and allocating resources in health regions: lessons from a project evaluating program budgeting and marginal analysis (PBMA). Health Policy 64(3):335–348CrossRefGoogle Scholar
  24. Nash R, Featherstone H (2010) Cough up: balancing tobacco income and costs in society. Policy Exchange, LondonGoogle Scholar
  25. Nutt DJ, King LA, Phillips LD (2010) Drug harms in the UK: a multicriteria decision analysis. Lancet 376(9752):1558–1565CrossRefGoogle Scholar
  26. Office for National Statistics (2012) Social Survey Division, Integrated Household Survey, April 2011–March 2012. UK Data Archive, ColchesterGoogle Scholar
  27. Perry A et al (2000) Measuring the costs and benefits of heart disease monitoring. Heart 83(6):651–656CrossRefGoogle Scholar
  28. Phillips LD (1984) A theory of requisite decision models. Acta Psychol 56(1):29–48CrossRefGoogle Scholar
  29. Phillips L (2007) Decision conferencing. In: Edwards W, Miles RF Jr, Von Winterfeldt D (eds) Advances in decision analysis: from foundations to applications. Cambridge University Press, CambridgeGoogle Scholar
  30. Phillips LD, Bana e Costa CA (2007) Transparent prioritisation, budgeting and resource allocation with multi-criteria decision analysis and decision conferencing. Ann Oper Res 154(1):51–68CrossRefGoogle Scholar
  31. Reddy B et al (2014) Prioritising public health guidance topics in the National Institute for Health and Care Excellence using the analytic hierarchy process. Public Health 128(10):896–903CrossRefGoogle Scholar
  32. Robinson S et al (2012a) Structures and processes for priority-setting by health-care funders: a national survey of primary care trusts in England. Health Serv Manag Res 25(3):113–120CrossRefGoogle Scholar
  33. Robinson S et al (2012b) Priority-setting and rationing in healthcare: evidence from the English experience. Soc Sci Med 75(12):2386–2393CrossRefGoogle Scholar
  34. Ruzante JM et al (2010) A multifactorial risk prioritization framework for foodborne pathogens. Risk Anal 30(5):724–742CrossRefGoogle Scholar
  35. Scott A, Currie N, Donaldson C (1998) Evaluating innovation in general practice: a pragmatic framework using programme budgeting and marginal analysis. Fam Pract 15(3):216–222CrossRefGoogle Scholar
  36. Sheppard SR, Meitner M (2005) Using multi-criteria analysis and visualisation for sustainable forest management planning with stakeholder groups. For Ecol Manage 207(1):171–187CrossRefGoogle Scholar
  37. Soma K (2003) How to involve stakeholders in fisheries management—a country case study in Trinidad and Tobago. Marine Policy 27(1):47–58CrossRefGoogle Scholar
  38. Twaddle S, Walker A (1995) Programme budgeting and marginal analysis: application within programmes to assist purchasing in Greater Glasgow Health Board. Health Policy 33(2):91–105CrossRefGoogle Scholar
  39. UK Border Agency (2010) Tier 2 of the points based system—policy guidance. Tier 2 version 04.10 04/10, H. Office, Editor. Home Office, LondonGoogle Scholar
  40. Phillips L (2007) Decision conferencing. In: Edwards W, Miles Jr RF, Von Winterfeldt D (eds) Advances in decision analysis: from foundations to applications. Cambridge University Press, CambridgeGoogle Scholar
  41. UK Government (2011) Option appraisal: making informed decisions in government. National Audit Office, EditorGoogle Scholar
  42. United Nations Framework for Climate Change (2002) Input of the least developed countries expert group on the improvement of the guidelines for the preparation of national adaptation programmes of action FCCC/SBI/2002/INF.14. United Nations, GenevaGoogle Scholar
  43. Von Winterfeldt D, Edwards W (1986) Decision analysis and behavioral research, vol 604. Cambridge University Press, CambridgeGoogle Scholar
  44. Williams I, Robinson S, Dickinson H (2011) Rationing in health care: the theory and practice of priority setting. Policy Press, BristolGoogle Scholar
  45. Wilson H, Scott S (1995) PBMA—its role in the future purchasing arrangements for health care services. Health Policy 33(2):157–160CrossRefGoogle Scholar
  46. World Health Organization (2003) WHO framework convention on tobacco control. WHO, GenevaGoogle Scholar

Copyright information

© The Author(s) 2016

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  • Brian P. Reddy
    • 1
  • Praveen Thokala
    • 1
  • Alison Iliff
    • 2
  • Kerry Warhurst
    • 3
  • Helen Chambers
    • 4
  • Lynsey Bowker
    • 5
  • Stephen J. Walters
    • 1
  • Alejandra Duenas
    • 6
  • Michael P. Kelly
    • 7
  1. 1.ScHARR, University of SheffieldSheffieldUK
  2. 2.Public Health EnglandLeedsUK
  3. 3.NHS England-North (Yorkshire and Humber)LeedsUK
  4. 4.Barnsley Metropolitan Borough CouncilBarnsleyUK
  5. 5.Yorkshire and Humber Academic Health Science NetworkWakefieldUK
  6. 6.IÉSEG School of ManagementPuteauxFrance
  7. 7.Institute of Public Health, University of CambridgeCambridgeUK

Personalised recommendations