1 Background

Smoking is the single biggest cause of preventable death in the United Kingdom (UK). Smoking is a major cause of coronary heart disease, some cancers and respiratory disease, including chronic obstructive pulmonary disease. These diseases have the greatest impact on the least well off in society and it is estimated that up to half the difference in premature death between the most and least affluent groups is explained by different patterns in smoking (Jarvis and Wardle 1999). Despite high tax rates on tobacco products, the total societal cost of smoking, in the UK, was estimated in 2010 at £13.74 billion (Nash and Featherstone 2010), representing a net loss to society of 6.5 p per cigarette smoked. In the “Healthy Lives, Healthy People: a Tobacco Control Plan for England” white paper (UK Government 2010), the UK government set out the ‘national ambitions’ in reducing smoking prevalence for the coming 5 years:

  • to reduce adult (aged 18 or over) smoking prevalence in England to 18.5 % or less by the end of 2015 (from 21.2 %), meaning around 210,000 fewer smokers per year;

  • to reduce the rates of regular smoking among 15-year-olds in England to 12 % or less (from 15 %) by the end of 2015;

  • to reduce the rates of smoking throughout pregnancy to 11 % or less (from 14 %) by the end of 2015 (as measured at the time if giving birth).

The metropolitan county of South Yorkshire is made up of four local authorities (Barnsley, Doncaster, Rotherham and Sheffield) and is home to approximately 1.3 million people. Each local authority acts independently and has its own director of public health (DPH). Each local authority has similar-sized budgets relating to tobacco and a tobacco control commissioner (TCC) responsible for the area. The overall smoking prevalence rates in the local authorities had not changed for a number of years, and these rates were higher at the time the project was initiated than the national average for England of 19.5 %: from 22.7 % in Rotherham to 25.3 % in Doncaster (Office for National Statistics 2012). This status quo was no longer felt to be tenable and a new approach was deemed necessary by the TCCs, who approached the School of Health and Related Research (ScHARR) at the University of Sheffield, hoping to identify an appropriate ‘knowledge-based approach’ that might inform future local strategies to reduce smoking prevalence in a rational and transparent way.

This paper aims to describe the use of a multi-criteria decision analysis (MCDA) to support health priority setting in local authorities to reduce smoking prevalence. A multi-criteria decision analysis approach was proposed and used, incorporating the views of the TCCs, the public, and an assembled expert and stakeholder group, to gauge a range of opinion, to present a range of policy options to each borough’s respective DPH. The next section describes the literature review conducted to identify approaches for local-level priority setting. Section 3 describes the MCDA methods used in our study and the discussion (Sect. 4) highlights the relative advantages or disadvantages of our approach. Section 5 presents the conclusions.

2 Literature review on priority setting at a local level

This section provides an overview of methods used to support health priority setting in local authorities. Some of these methods have been developed in health care independent of MCDA, but share a number of similarities. The priority setting approaches that drawn on MCDA techniques, including the Portsmouth Scorecard, Option Appraisal, Socio-Technical Allocation of Resources (Star), as well as Programme Budgeting and Marginal Analysis (PBMA).

In the Portsmouth Scorecard, each option is scored against selected criteria and total scores are calculated and discussed. A single index score is calculated for each intervention with the highest scoring interventions taking priority (Williams et al. 2011). This approach is easy to understand by decision makers and the criteria can be adapted to suit the priority setting context (Robinson et al. 2012). Different versions of the Portsmouth Scorecard have been developed to inform decisions over clinical and non-clinical interventions in a number of localities across England (Robinson et al. 2012).

Option Appraisal (Brazier 1987; Akehurst et al. 1988), another simple decision analysis technique, has been in use since the 1980s. It involves defining objectives, examining options and weighing up the costs, benefits, risks and uncertainties (HM Treasury 2003; UK Government 2011). Examples of Option Appraisals include mental health services in Ashton, Leigh and Wigan (2005) a well as Aberdeen City Council (2010) and coronary heart disease monitoring in Scotland (Perry et al. 2000).

Airoldi et al. (2014) used MCDA techniques to structure a prioritisation process at the Isle of Wight Primary Care Trust (PCT), which has since been developed into the Socio-Technical Allocation of Resources (Star) toolkit in collaboration with the Health Foundation. This method comprises decision conferencing for stakeholder engagement and a value for money analysis. An Excel-based tool is used alongside a facilitated stakeholder workshop and the results are presented using visual aids such as value-for-money triangles and efficiency frontiers. Examples of Star include the Isle of Wight (Airoldi et al. 2014) and Sheffield PCT (Williams et al. 2011; Airoldi et al. 2014; Airoldi 2013; Health Foundation 2012).

Programme Budgeting and Marginal Analysis (PBMA) consists of identifying resources and how they are being spent in specified programmes (PB) before considering the expected incremental costs and effects of the proposed investments/disinvestments at the margin (MA), to maximise benefit and minimise costs (Wilson and Scott 1995; Donaldson 1995; Madden et al. 1995). A broad framework of PBMA is presented by Mitton and Donaldson (Mitton and Donaldson 2003), which has been applied for example in gynaecological services (Twaddle and Walker 1995), diabetes care in general practices (Scott et al. 1998), respiratory disease (Bohmer et al. 2001) and coronary heart disease (Haas et al. 2001).

MCDA approaches have been used in a wide range of public health settings at non-local authority level such as investigating government drug policy (Nutt et al. 2010), shared decision making to help choose the most appropriate screening procedure for a patient (Dolan 2000) and estimating the risk from food-borne diseases (Ruzante et al. 2010). They have also been widely used in non-health settings, especially those related to environmental contexts. The United Nations has used it in a number of areas, such as climate change (United Nations Framework for Climate Change 2002) and fisheries management (Soma 2003); others have used it for areas such as water safety (Lindhe et al. 2013), forestry (Sheppard and Meitner 2005) and waste management (Hanan et al. 2013). It has also been used in UK public sector settings outside of health to provide yes/no decisions, such as the UK Border Agency’s point-based system on whether to allow citizens from outside the European Economic Area to work and settle in the UK (UK Border Agency 2010). However, a key aim of this study was to design an appropriate model for use at a local government level, given the specific problem at hand.

There are a number of different MCDA methods for priority setting. Issues of availability of data, the very purpose of the approach, its next steps, and availability of time to carry out the process are important in making final decisions regarding which MCDA approach to use and which compromises might be necessary and acceptable. The sophistication of the method can range from simple methods (e.g., Portsmouth Scorecard) to more complex methods (e.g., PBMA) as shown in Fig. 1. Another aspect of consideration for priority setting is the availability of evidence. As seen in Fig. 1, data on the alternatives’ performance on each of the criteria can be gathered in a variety of ways, ranging from evidence synthesis to expert opinions.

Fig. 1
figure 1

Field showing range of possible MCDA-based approaches, described by evidence requirements and resource/time use

Full size image

Whilst these applications in Fig. 1 are all based on MCDA principles, they vary significantly in terms of scientific rigour, robustness and their requirements regarding participants’ time and other resources. The MCDA method used in our study, which we have called the South Yorkshire Multi-criteria Decision Analysis Prioritisation for Local Effectiveness (SYMPLE) was chosen based on the priorities and constraints relating to resources and time, as well as scientific validity and the significance and broader context of the decision problem.

3 Methods

This section describes the details of the MCDA approach used in our study. There were three distinct phases to (which are split into 12 stages) in our MCDA approach:

  1. 1.

    Problem structuring.

  2. 2.

    MCDA modelling.

  3. 3.

    Supporting decision making.

The 12 stages are listed in Table 1, with the three phases above separated by dashed lines. The sequence of stages allowed the views of the general public, expert opinion and other stakeholders to be incorporated at various stages. The findings were ultimately used as a basis for prioritisation among the options under consideration. The stages were meant as a rough guide and revisited as necessary throughout the process—the entire process was therefore best seen as a framework for investigating and discussing the objectives and issues rather than a rigid formula.

Table 1 Involvement by relevant groups in project’s stages and phases
Full size table

3.1 Problem structuring

This stage required explicit articulation the aims of the process, the list of stakeholders potentially impacted and who should be responsible for making the final decision. This needed to be designed cognisant of the broader decision-making context and the potential consequences of using such a novel technique in this setting, given that there would be inevitable winners and losers from any decision. Consideration of both scientific/technical and social/human factors using a ‘socio-technical’ approach (Dodgson et al. 2009) is something of an art, but fundamental to the success of such a process.

3.1.1 Stakeholders

The TCCs identified an expert steering group made up of a wide variety of stakeholders to support the prioritisation process. These stakeholders came from a range of relevant settings with a role to play in reducing tobacco consumption, including specialities such as current stop smoking providers, midwifery, elected councillors, the fire service and a local chamber of commerce. This allowed for a wide range of views to be included and potentially increases the legitimacy of the findings, as practically all relevant parties were represented in the process.

3.1.2 Alternatives

Fifteen interventions were examined, chosen to represent a broad range of potential techniques. Five of the six strands advocated by the World Health Organisation (2003) and the UK government (All-Party Parliamentary Group on Smoking and Health 2010) to overcome practical barriers in tobacco control were included; the only such theme not represented was reducing tobacco advertising and promotion, which was felt to be outside the scope of local government control. Stop smoking support (SSS) services—which previously constituted around 90 % of tobacco spending in each local authority—were investigated alongside other interventions to ensure increased clarity and in the hope that a more effective holistic approach might emerge. There are potentially a limitless number of interventions available at various levels of investment, but MCDA techniques could not investigate every such intervention in practice. Given the findings would still be subject to political factors, the list was intended to be broad and representative rather than comprehensive. Three- to four-page business cases for each intervention were prepared, giving a broad outline of how the intervention would likely be implemented in practice and written in language accessible to all stakeholders. Brief descriptions of the interventions are available in the appendix. The interventions considered (grouped by the WHO ‘strands’) were:

  • targeted stop smoking support (SSS) services:

    • for routine and manual socioeconomic groups (R&M);

    • for people with long-term conditions;

    • for parents;

    • for black and minority ethnic communities (BME);

    • for people with mental health conditions;

    • pre-operatively;

    • in secondary care in hospital;

    • in maternity settings;

  • reducing exposure to secondhand smoke;

    • extend smoke-free spaces beyond legislation by voluntary extensions into areas families use;

    • SoNIC (“Social Norms in Local Communities”) programme to influence behaviour change;

  • media and education campaigns;

    • mass media, population-based communications and marketing campaigns;

    • customised, ongoing children and young people education programme;

    • replication of “A Stop Smoking in School Trial” (ASSIST) programme;

    • training for public sector staff, businesses and communities to provide brief advice to smokers and those affected by smoking;

  • tobacco regulation/reducing availability of tobacco;

    • enforcement to enhance the statutory functions of the local authority to enforce legislation on under-age sales and the selling of illicit tobacco.

3.1.3 Choice of the MCDA method

A standard linear additive MCDA model was chosen as the most appropriate approach for this decision problem. This is the most commonly used approach in health care (Marsh et al. 2014). Swing weighting was used to weight the criteria as it allows for the trade-offs which are inevitable in public health decision prioritisation (Baltussen and Niessen 2006) and because such weighting fulfils the theoretical requirements of MCDA (Belton and Stewart 2002). Scoring was performed using direct rating as they were elicited from the expert group. An emphasis was placed on achieving consensus by bringing stakeholders together at a meeting to debate and score topics, thus encouraging their understanding of the final decision and increasing their ‘buy-in’ (Phillips 2007). As a result, the ‘social’ aspects of the socio-technical process at the meeting had to be carefully managed (and are further described in Sect. 3.2.3 below). Approaches that are typically carried out by individuals, such as DCE—rather than those that required discussion and cooperation—were not considered appropriate on this basis. Only one-half day was available to bring together the expert group together. Thus, we chose a simple approach. Previous experience had suggested that the analytic hierarchy process may be too slow moving for such a group (Reddy et al. 2014) and was therefore not used. The total scores were calculated for each intervention, using a simple weighted sum method, which was then used to prioritise among the interventions.

3.2 MCDA modelling

To create a complete ranking, stakeholders had to specify the decision’s relevant criteria (and their relative importance) and the performance of the alternative interventions on each.

3.2.1 Criteria

The draft list of suitable criteria was identified through public consultation. Workshops were set up to engage with volunteers from the public to discuss a series of potential interventions and describe why they would prefer investment in one rather than another. These criteria were ranked according to participants’ views. They were then sent out to the expert group for comment and ranking and were subsequently revised. These steps were non-binding, but were used to inform the subsequent debate on which criteria to use.

A number of iterations were required to arrive at the final workable set of criteria. Some criteria likely to be highly related had to be removed or revised to avoid double counting, which might have biased the results. Others, such as feasibility, were removed as all alternatives under consideration were a priori felt to be similarly implementable. This process took some time, but once consensus was reached it was felt to have been an extremely valuable exercise in building a shared understanding of what makes an intervention worthwhile and in identifying underlying goals of the tobacco programme across all the strands. Overall six headline criteria were identified, as shown in Table 2. Given difficulties in generating (and subsequently communicating) criteria and their weights by correspondence, these were by necessity defined quite broadly, as themes to be considered as part of the discussions on the day of the workshop.

Table 2 Inputs to MCDA process prior to decision conference: the criteria, and their weightings
Full size table

3.2.2 Weighting

The preference weights for the criteria elicited using swing weighting are also shown in Table 2. These synthesised the viewpoints of the expert panel, the TCCs and the level of difference between the best and worst performing intervention on each criterion. Many of the expert panel expressed a view that large impacts to individuals should be preferred to interventions that have small impacts to larger numbers, and that equity aspects should be considered from the beginning (rather than after cost-effectiveness analyses had been carried out). In contrast, the main concern of the TCCs was the prevalence rates described in the national priorities. The importance of each criterion was elicited by correspondence before the meeting of the expert group due to expected time constraints on the day. These were converted into swing weights by the lead researcher based on deliberation with the TCCs regarding the range from best- to worst-performing intervention on each criterion. While not ideal, this was felt to be a necessary compromise. Sensitivity analyses were later used to ensure that the results were meaningful, in light of the fact that the expert group was not always in complete agreement on weightings; these weightings were found to be robust.

3.2.3 Scoring

The next stage required scores to be attributed to each intervention on each criterion. Due to the scarcity of evidence of the interventions on the criteria, it was felt that the subjective opinion of the key experts should be used to elicit the scores. This was carried out in a ‘decision conference’ setting (Phillips and Bana e Costa 2007). To do so, an impartial facilitator worked iteratively with stakeholders to generate an explicit model intended to help those present to think more clearly about the relevant issues. Care was taken to ensure that as broad a range of potential stakeholders were included in the process as possible. About 30 participants from the expert stakeholder group took part, representing the range of specialities from inside and outside health care, and chosen from across the four boroughs.

Participants were initially split into four groups, to ensure that all participants would have the opportunity to voice their opinion at each stage. Participants were assigned according to speciality and borough to ensure a similar mix on each table. They were asked to consider how an intervention was likely to perform on each criterion, in batches of four interventions at a time, and agree upon a consensus score from 0 to 10 based on these discussions, where 0 indicates having the least imaginable impact for that criterion and 10 indicated the biggest imaginable impact (and other intermediate descriptions provided). Tables then fed back their scores to the floor, and debate could occur between groups to ensure an overall consensus score for each intervention on each criterion that emerged. Due to time pressure towards the end of the day, two larger groups were formed to score the final interventions in parallel. The final scores elicited for the interventions are presented in Table 3.

Table 3 Consensus scores generated at decision conference for each intervention on each criterion
Full size table

3.2.4 Aggregation

Scores elicited from participants on each intervention were combined with the weight on each criterion to calculate the total scores for each intervention under consideration. This was performed using a weighted sum approach as shown below:

$$V_{\text{j}} = \sum\limits_{0}^{i} {s_{ij} \times w_{i} } ,$$

where V j is the overall value for intervention j estimated from MCDA model, s ij is the score for intervention j on criterion i and w i is the weight attached to criterion i.

It is important to reiterate that the primary result of the model should be to aid the process of structuring and understanding the decision, rather than necessarily providing some ‘correct ranking’ (Phillips 2007). MCDA practitioners often speak of the importance of parsimony or ‘requisiteness’ in such circumstances (Phillips 1984). The ultimate responsibility to consider all relevant factors (including those not contained within in the model) rested with the relevant directors of public health. Nonetheless, the interventions could produce initial rankings based on their total scores, as shown in Table 3.

3.3 Supporting decision making

Before making their final decisions, the directors of public health (as advised by the TCCs) needed to be confident in the results of the approach and to better understand the robustness of the findings. The following stages investigated the applicability of the results for use in practice.

3.3.1 Examining the results

The results implied that increased coordination with local authority trading standards teams, as advocated in the “Enforcement” intervention, best meets the range of issues relevant to the concerned local authorities. Maternity stop smoking services and investment in communications and marketing, amongst others, also performed well. A number of approaches could potentially be used together in the future. A school-based programme for teenagers, entitled ASSIST, performed very poorly as did Stop Smoking Services relating to hospital inpatients and mental health. Differences in interpretation of some interventions (some participants at the meeting felt it would be infeasible or counter-productive to attempt to ban smoking in parks for example) meant some alternatives were given two sets of scores. Scores were reported separately for the different groups based on whether they thought the interventions offered high value for money (VfM) or low VfM, though this did not change the results notably. Table 3 lists these differing potential interventions independently of each other, to allow comparison.

3.3.2 Sensitivity analysis

The MCDA process is only a decision support tool and it would be inappropriate to take the rankings as a ‘correct answer’ without further examination. We performed sensitivity analyses to investigate the robustness of the findings, which required investigating by how much the weighting on each criterion would have to change to change the final ordering of interventions. These are difficult to illustrate here across 15 alternatives on 8 criteria, but, generally speaking, the magnitude of change in weighting required to alter rankings implies the results are stable. The only exceptions were for those interventions with equal total scores [smoke-free spaces (private spaces only); mental health stop smoking support (high value for money); and preoperative stop smoking support services] for whom any changes in weighting naturally changed their ordering. These three interventions performed relatively poorly anyway and were unlikely to be funded. The smallest change to alter the rankings was if the weighting on reducing prevalence amongst adults was reduced from 20 % of weighting to 18.75 %, maternity SSS overtook it as leader. Because these were the two highest ranked criteria, both were likely to be provided regardless, meaning in practice that the model is in practice more robust than even this. The robustness of the findings is typical in MCDA techniques (Von Winterfeldt and Edwards 1986) and also explains why for different interpretations from the panel (and hence different scores) for certain criteria, the scores attributed had little overall impact on rankings.

3.3.3 Implementing findings

The results firmly show that enforcement, maternity SSS and preventative measures (except ASSIST) appear to perform well in all circumstances according to these criteria. The four councils have reduced the overall spending on SSS services, and realigned spending to focus more on pregnant women in particular. Spending has increased across the board on enforcement, with at least two councils also hiring enforcement officers—which is all the more impressive given the widespread staff cuts. Barnsley also hired a stop smoking midwife on the strength of the report. Education campaigns for young people have been prioritised as a preventative measure. Sheffield has also increased spending on smoke-free spaces, and at the time was the only public health topic to have its budget protected.

4 Discussion

The paper has described and demonstrated that an MCDA approach can be used to support health priority-setting in local authorities to reduce smoking prevalence. The MCDA approach used an open, deliberative process, bringing together experts and stakeholders to rank interventions and making explicit any assumptions. This was the vehicle by which an initial impasse over appropriate areas for outcomes of tobacco spending was transformed into tangible policy options, addressing a range of relevant issues. The open and transparent nature of the approach allowed public and expert opinion to be used at various stages of the process as required, increasing the likelihood of political acceptability and potentially increasing the fairness of final decisions (Daniels 2000). The approach achieved more than just scoring potential actions; it empowered all four councils to make significant changes in the services commissioned and served as something of a uniting mission statement for the project.

It was felt by the TCCs a priori that the results must show legitimacy to be implementable and hence a broad range of stakeholders and public viewpoints would need to be included as part of the process. This ensured that a variety of perspectives are accounted for and that the stakeholders’ (including both ‘winners’ and ‘losers’) own buy into the process, which is important in ensuring successful implementation (Phillips 1984). The large number of stakeholders involved created challenges in terms of communication, which made the process more disjointed than would be ideal. The generation of criteria, weighting and scoring therefore had to be carried out separately and subsequently sewn together by the lead researcher and TCCs. It also impacted on the length of time available at the decision conference, influencing the choice of the MCDA approach used. Criteria were also by necessity defined broadly, though at the meeting the expert group (who had played a major role in devising them) seemed broadly in agreement as to their purpose and relative importance. Sensitivity analyses found these to be robust, and this stage is vital in such circumstances where there is the risk of confusion or disagreement.

Participants and DsPH seemed happy with the results; however, there were some counter-intuitive findings. Most notably, ASSIST came last on the list, despite it being one of the few available interventions for prevention of smoking supported by sufficient evidence of effectiveness that it is recommended by the National Institute for Health and Care Excellence (NICE). It was felt by the expert stakeholders to be unsuitable, given the local environment (especially, due to the perceived risk of schoolyard bullying). As the local government has assumed extra powers since the passage of the Health and Social Care Act, 2013, this may lead to similar conflicts in future between national and local-level decision makers, both in terms of relevant evidence and local preferences.

The results of this approach were only one stage as part of a political process and would only inform the ultimate decisions of the respective directors of public health. The model’s recommendations therefore had to jump another hurdle to ensure political feasibility and common sense. The DsPH could also consider whether there were synergies among interventions (and indeed if further savings were possible by collaborating with other boroughs), neither of which was considered in our process.

Each stage of our study was designed pragmatically to ensure that it had the legitimacy of a wide range of stakeholders and could give a good overview of the likely impact of the interventions under consideration, while taking into account a range of relevant constraints in doing so. Though the approach chosen cannot necessarily therefore be said to conform to ideal best practice, such compromises are often necessary in real-world settings. The programme emerged from a situation where clear evidence of effectiveness was impossible and circumstances dictated that insights would have to be drawn from expert evidence. We attempted to source this in an as systematic a manner as was possible in the circumstances given the context. Ensuring that such context is taken into account was fundamental in ensuring the success of the project.

5 Conclusions

An MCDA approach was used to reappraise public health spending on reducing the harm caused by tobacco across the four local authorities making up South Yorkshire. The ability of MCDA techniques to incorporate data as well as both qualitative and quantitative judgements in a formal manner means that they are well suited to support public health decision making, in which evidence is often only partially available and many policies are value driven. A number of MCDA methods exist for local resource allocation, ranging from quick and simple methods such as the Portsmouth Scorecard to more sophisticated methods such as PBMA. The local decision makers need to choose the appropriate method based on their resource/time constraints, scientific validity, and the significance and broader context of the decision problem.