FormalPara Key Points for Decision Makers

Our stakeholder engagement programme provided key learning on policy context, established working relationships, informed the conceptualisation of the model, and provided expert advice on critical assumptions and data sources throughout the model building process.

Integrating stakeholder engagement into the value-laden development process and communication of a public health economic model can improve the contextual relevance, credibility and potential for impact of the research.

1 Background

Health economic modelling of public health policies can contribute significantly to policy development, and there is a growing acknowledgement of the need to bridge the gap between academics and policy makers [1, 2]. Essential to achieving this is for researchers to work more closely with expert stakeholders, including those with expertise in policy, modelling, epidemiology and local data, and those who experience the impacts of the public health issue in question.

Public health decision makers generally have a broader set of policy goals than those operating in other parts of the health system, where health maximisation is the predominant focus of cost-effectiveness modelling. For example, this might include a concern for equity and subsequent interest in the distributional impacts of a policy. These interests tend to be defined outside of academia and draw instead on socio-political considerations. Incorporation of these often value-laden goals within economic modelling should be informed by societal values, necessitating modeller engagement with diverse communities of stakeholders [3, 4].

In addition, the complex nature of public health economic models is such that there are a variety of epidemiological model structures available that go beyond those routinely used in health technology assessment [5]. This can leave the researcher with many important structural modelling decisions that would benefit from the input of domain experts [6].

While engaging with stakeholders enables a research team to seek input for a number of study design decisions, for example which subgroups are the most important or what outputs should be generated, it is important to acknowledge the modelling team bring their own beliefs and values to the work. Consequently, the process of stakeholder engagement, including whom to engage, what topics to cover, and how the responses are interpreted, is inherently value-laden and not always a straight-forward process.

This paper provides an example of a programme of stakeholder engagement undertaken as part of research modelling the impact of minimum unit pricing of alcohol in South Africa [7,8,9], with two main objectives: Firstly, to shape the direction of the research using expert local knowledge (including understanding the problem, guiding model development, and ensuring face validity). Secondly, to provide channels for communication vital for increasing the potential for impact. The engagement was split into four phases: developing a detailed understanding of the local policy context; co-producing the model focus and structure; scrutinising model development and communication planning; and communicating research evidence to end-users. We outline the methods for engagement, inputs received, and decisions taken at each of these stages.

2 Methods

2.1 Overview

The overarching research project aimed to build a public health economic model for an alcohol pricing policy in South Africa, with the ability to estimate how the impacts differ by equity-relevant subgroups. This research aimed to contribute to an evidence base providing potential policy options to reduce the significant, and unequally distributed, burden of alcohol-related harm in South Africa.

The stakeholder engagement can be characterised by four phases fully integrated with the model development, namely: developing a detailed understanding of the local policy context; co-producing model focus and structure; scrutinising model development; and communicating research evidence to end-users. Following a stakeholder mapping exercise, scoping interviews, workshops and ongoing ad hoc communication facilitated the process (Fig. 1).

Fig. 1
figure 1

Project overview

Full size image

All materials, including interview questions and workshop presentations and activities, were piloted with peers (academic researchers) before delivery. Special attention was given to avoiding the use of technical language to promote engagement.

2.2 Stakeholder Selection

A stakeholder mapping exercise was carried out [10] using academic papers reviewing alcohol policy development in South Africa, an economic assessment of proposed alcohol legislation by a private consultancy [11,12,13,14], and consultation with three experts from three South African academic institutions (University of the Western Cape, University of Witwatersrand, and University of Cape Town). We identified ten categories of stakeholders with a potential interest in alcohol pricing policy: policy makers and government; general public; civil society organisations; practitioners and professionals; lobby groups; media; international public health bodies; research community; private business; and police (specific entities under each of these categories are given in Appendix 1 in the Online Supplementary Material (OSM)). This mapping exercise provided familiarisation with high-profile experts/stakeholders/organisations in the field and a list for future research dissemination.

Following this, we drew up a short list of experts working in alcohol policy in South Africa selecting just three of the stakeholder categories: policy makers and government (both local and national); the research community; and civil society organisations. We chose stakeholders from government so that our research would be tailored to important policy questions and evidence gaps, and to establish working relationships that would provide opportunities to communicate our research directly to those who would be developing alcohol policy. We chose researchers (academics) in order to expose our work to critical questioning from those with expertise in local data and modelling methods already applied in South Africa. We chose civil society organisations as we believed them to be close to those who suffer the impact of alcohol harm in South Africa while still having the professional skills to immediately be able to engage and contribute to the research. The civil society groups provided an important check on the modelling team, as well as the other stakeholders, who may be disconnected from how alcohol harm affects the poorest groups. For example, the members of one organisation were residents of an informal settlement in Cape Town and were able to describe the activities of alcohol companies in their township and the impact of heavy drinking. They also provided helpful checks on some of the data, for example the price of alcohol in shebeens (unlicensed alcohol premises in informal settlements).

There were many other groups that we did not engage, including business, the media and general public. This was a pragmatic decision taken within the constraints of the project. Specifically, we wished to avoid the management of a disparate set of conflicting interests within a highly politicised policy area, which were also expected to make recruitment of the other stakeholder groups more difficult. We also were not able to devote more resource to engaging groups or individuals unused to being consulted on policy and research.

2.3 Phase One: Developing a Detailed Understanding of the Local Policy Context

Phase one aimed to develop our understanding of the policy context in which the alcohol pricing policy would be potentially implemented, and increase our awareness of high-profile alcohol harms and policies, key stakeholders, and alcohol-related inequalities and inequities.

We conducted 12 face-to-face semi-structured interviews. The questions focused on three areas: alcohol harm in South Africa, pricing policies and equity (Appendix 2, OSM). The conceptualisation of equity was kept broad to provide examples of how the term is understood rather than to answer specific modelling questions more suited to the workshops. To limit participant burden, all interviews lasted between 30 and 60 min. The interviews were carried out face-to-face, six online and six in person in Cape Town, South Africa.

Three of the interviewees were from policy-setting organisations (National Treasury, Department of Health, Western Cape Liquor Authority), five were from civil society organisations (DG Murray Trust (2), South African Alcohol Policy Alliance, Khayelitsha Health Forum, Violence Prevention through Urban Upgrading), and four were academics engaged in alcohol policy and research (University of Cape Town and South African Medical Research Council).

2.4 Phase Two: Co-producing Model Focus and Structure

Phase two aimed to inform core modelling decisions including the choice of policy, sub-populations and outcomes. Following introductions at the start of the workshop, we presented the research overview (Fig. 1 above) emphasising how stakeholder inputs shape the research throughout.

2.4.1 Objective One: Developing a Common Understanding of the Public Health Problem

Participants were split into three small groups purposively chosen to ensure policy and academic and civil society representation. As a modelling team our shared belief was that alcohol use in South Africa presents a public health problem that warrants intervention, and that sufficient evidence exists to support pricing policies as a potentially cost-effective policy option. The specific objective of this exercise was to inform a problem-orientated conceptual model of the burden of alcohol in South Africa. This technique draws upon frameworks utilised within operational research for knowledge elicitation and brings together the stakeholder and researcher’s understanding of the problem in an accessible form [6, 15,16,17,18].

A simplified map depicting causal pathways between consumption and alcohol harm in South Africa was presented and explained before each group took an A0 hardcopy to edit (Appendix 3, OSM). After 30 min the edited maps were presented back to the whole group. All changes were later merged onto one new diagram and further revisions added after email circulation post workshop.

2.4.2 Objective Two: Choosing the Policy Intervention to Model

The modelling team had decided and communicated in advance of the workshop that alcohol pricing policies, and in particular minimum unit pricing options, would be considered as from prior engagement we knew these to be of interest to South African policy makers. However, any model would be capable of evaluating other price-based policies, and so the team wanted to ask stakeholders what other pricing policies were of interest. This was also felt to lead to more engagement from the group and a fuller examination of the differing impacts of the different policy instruments.

Stakeholders attended a presentation about the current tax system and were then asked to consider the following options: Keep current system but increase rates; introduce a volumetric tax system; introduce minimum unit pricing; any other policy they suggest; a combination of policies. They split into small groups for discussion, then opinions were fed back to the whole group.

2.4.3 Objective Three: Choosing Sub-populations of Interest

We facilitated a whole group discussion to capture all sub-population groups that might be of interest. Groups were written on a flip chart until saturation point was reached. Further group discussion involved narrowing down the list to key groups of interest, acknowledging potential data constraints in relation to the associated policy model.

2.4.4 Objective Four: Choosing Model Outcomes

We facilitated a group discussion, firstly outlining which outcomes would be included in the model as a minimum, and then asking the group to suggest other important outcomes. Suggestions were written on a flipchart. Once saturation point was reached everyone was asked to privately write down three outcomes they considered most important. Participants were then given stickers and invited to come up to the flipchart and place their stickers against the three outcomes they had chosen. The order of priority for including outcomes in the model was determined by the number of votes against that outcome.

Following the workshop, all participants, and those who had expressed an interest but were unable to attend, received an outputs document via email detailing all the key decisions that had been made and inviting feedback.

2.5 Phase Three: Scrutinising Model Development and Communication Planning

Phase three centred on a second workshop with three objectives: to provide a progress update, to present preliminary results, and to create research communication plans.

Due to COVID-19 the workshop was moved online, with the original plan redesigned. Live polls, online short questionnaires and a chat box were used to maintain engagement and provide written outputs for analysis. A significant proportion of time was given for participants to share their name, organisation and interest in the topic in order to promote a sense of connection.

2.5.1 Objective One: Presenting Model Structure, Data and Assumptions to Allow for Scrutiny

Two 10-min presentations were given, each followed by 10 min to fill in an online questionnaire. Pragmatically we could not consult on every detail of the model so prioritised questions about data sources and assumptions that we believed had the potential to make big differences to the results (e.g., baseline consumption and elasticities) and those relating to the specific cultural context with which we were less familiar (e.g., switching to homebrew). The first presentation focused on the price policy to consumption section of the model. We presented and asked for feedback on proposed data sources and estimates for baseline consumption levels, drinker type categories (heavy, binge, moderate), baseline alcohol prices and elasticities. We also asked for input on key assumptions such as whether those who start as non-drinkers should be able to transition to drinking after a policy, whether drinkers may switch to homebrew as a result of the policy, and, if so, to what degree. The second presentation focused on how we proposed to model the relationship between alcohol consumption and health harm. Again, we presented the data sources and related assumptions we intended to use for baseline prevalence, dose-response relationships, multipliers for prevalence to hospitalisation, and health unit costs.

2.5.2 Objective Two: Presenting Preliminary Results

A presentation of preliminary results was given. This was partly for the interest of the participants but also crucially to inform the next session on how model results should be communicated. A number of live polls were used throughout the presentation, primarily to maintain engagement but also to gather additional feedback. Participants were asked whether the pie charts were helpful, whether the bar charts were understandable, and whether comparing different levels of minimum unit pricing (MUP) as part of the results output was useful.

2.5.3 Objective Three: Planning Future Research Communication

Finally, we asked how to communicate results. We provided an online table and asked people to populate it with suggestions for the target audience, methods of communication, and important topics. As with the first workshop, an outputs document was circulated within a week of the session and feedback invited.

2.6 Phase Four: Communicating Research Evidence to End-Users

The final workshop, also online, was written primarily for a policy audience (as determined by stakeholders in the previous workshop) and focused on presenting the results of the research project. The objectives were to explain the pricing policy, outline the modelling methods used, present the final results, and allow for a discussion about future policy and research direction in light of the evidence presented.

In an effort to promote a productive discussion, we asked three stakeholders in advance if they would share their reflections on the research. They were drawn from national government, local government and a civil society organisation. We gave them advance sight of our presentation slides and asked for reflections on future research and policy direction. The discussion was then opened to all participants.

2.7 Follow-Up Communication

Slides were circulated after the event. Following publication in an academic journal [8], the results paper was circulated accompanied by a one-page summary written from a policy-making perspective (Appendix 4, OSM).

3 Results

Here we highlight how the four phases of engagement directly shaped the modelling and led to opportunities for research impact. A list of organisations attending at each of the engagement activities is provided in Appendix 5, OSM.

3.1 Outcomes of Phase One: Developing a Detailed Understanding of the Local Policy Context

The scoping interviews enabled initiation of positive working relationships and increased understanding of the alcohol policy landscape. We learnt which alcohol harms stakeholders were most concerned with (namely HIV and risky sexual behaviour, gender-based violence, road traffic injuries, and fetal alcohol syndrome) and which policies they knew of. We gathered data on their knowledge of the current tax system and attitudes to MUP. Broader concerns around equity, including in relation to race and the poor or economically disadvantaged, were identified. These interview data informed both the broader research project and the next phase of engagement in important ways.

Firstly, we learnt that understanding the supply chain relating to the alcohol industry, and how to ensure MUP is enforceable, would be crucial to the success of MUP. Although modelling the supply chain and alternative market responses was beyond the scope of the research, due to time and data constraints, the knowledge allowed us to engage in these discussions when delivering our final presentation to stakeholders.

There were several normative issues raised by the stakeholders. Firstly, it was clear that modelling scenarios around substitution effects for homebrew were viewed as important given that homebrew is primarily drunk by poorer groups, a subgroup of focus amongst our stakeholders. Secondly, it was highlighted that opponents to pricing policies may use arguments around black economic empowerment to oppose any additional pricing restrictions; this did not influence the development of the model, but we were aware our results might provide information relevant to the broader policy debate. Lastly, the economic cost of alcohol to society and a concern for both health and financial outcomes for the poor were consistently seen as important. Our stakeholders were concerned about cross-sectoral impacts, citing the wide-reaching impact of alcohol on crime, education, productivity and health.

In terms of adapting how we engaged with the stakeholders, we discovered the term ‘equity’ needed to be dropped for being too contentious and understood in a multitude of ways; instead we used ‘differential impact’ or ‘distributions’. It was also clear people were unfamiliar with the tax system and would need an explanation, including the point that MUP would not replace taxation. We found stakeholders connected with international comparisons, particularly for MUP (Scotland and Russia), and so we integrated examples into future presentations. Finally, we noted that people were interested in the background and expertise of the modelling team and therefore we planned sufficient time for introductions at the first workshop.

3.2 Outcomes of Phase Two: Co-Producing Model Focus and Structure

The first outcome was the problem-oriented conceptual model, which created a shared understanding of alcohol harm in South Africa within the group and with us, as the research team. The stakeholders then went on to provide critical modelling decisions. They chose MUP as the policy to model, over alternative pricing policies, which were also discussed. There was disagreement between stakeholders as to what level the MUP should be set at. A discussion was held in which a range was agreed upon; these levels were open for review at subsequent workshops. Stakeholders indicated the model should allow for the following sub-populations: age, sex, drinker type (taking into account both mean and peak alcohol consumption), income and potentially urban/rural differences/status. Finally, stakeholders judged the inclusion of government costs to be a high priority alongside employment, crime and lost earnings. The most important health outcomes were indicated as violence and HIV. At the end of this phase participants noted that they would like to see more policy professionals involved in the research.

3.3 Outcomes of Phase Three: Scrutinising Model Development and Communication Planning

The data source used to estimate dose response relationships [19] was presented, and stakeholders agreed with the modelling team that is was appropriate; alternative sources did not include HIV as attributable to alcohol, a key area of focus. One stakeholder, with expertise in South African epidemiological data, challenged the use of Global Burden of Disease (GBD) estimates [20] for baseline prevalence, citing a letter to the Lancet that claimed GBD overestimates HIV deaths for South Africa [21]. National Burden of Disease (NBD) estimates were suggested as preferable [22]. We spent time interrogating the data and found the latest GBD figures were broadly in line with national estimates. There would be two major drawbacks to using the NBD study. Firstly, it only covered mortality and not prevalence. Assumptions would need to be made to estimate prevalence. Secondly, it dates back to 2012. Considerable work would be needed projecting the population forward, incorporating changes to the age structure as well as disease trends, all of which would be subject to error. The same stakeholder, who recommended the NBD estimates, informed us that the next national figures would not be available until 2022, and so a decision was made to use our originally suggested data source. This decision and justification were communicated to this specific stakeholder, and were accepted.

Stakeholders agreed that the alcohol prices used were plausible for South Africa (14 said yes, and two said maybe). Those who said ‘maybe’ provided questions around whether richer people pay higher prices. This appeared likely but was not supported by our dataset and was written into the limitations section of our research. Stakeholders indicated that we should model switching behaviour to homebrew for those who drink both homebrew and recorded alcohol, when drinkers are faced with a price rise as a result of the policy.

Stakeholders with expertise in modelling suggested that our categorisation of drinkers by consumption behaviour should be reviewed and justified. In particular, it was suggested that the category defined as heavy drinkers, according to our criteria, would include regular binge drinkers as well as those who drink just two drinks three times a week (labelled by our stakeholders as intermediate drinkers, distinct from moderate drinkers who drink less than six standard drinks per week). It was suggested these groups were not homogenous and thus would respond to prices differently. We explored a number of different options for splitting the categories but found that our data could not support a group of intermediate drinkers, there were simply too few. We found that 95% of those who we had classed as heavy drinkers were also binge drinkers. Interrogating the data was carried out alongside an ongoing dialogue with these stakeholders.

Of the 14 stakeholders still present (two stakeholders did not attend the whole session) ten indicated that the use of UK data (namely multipliers to convert prevalence of a disease to hospital admissions) to replace limited local older data would reduce the credibility of the model. Therefore, we chose to use a published South African academic paper to calculate multipliers.

They indicated that the modelling should prioritise health outcomes, although crime was also important. They considered the costs already included to be relevant but also highlighted the importance of costs to household expenditure, lost wages and reduced productivity. As the modelling did not include all societal costs, stakeholders felt it important to highlight in any communication that the research only investigates a small proportion of the impact of alcohol in South Africa.

Online workshop attendees agreed that the pie charts and bar charts presented were understandable. Results demonstrating the impact of different minimum price levels were also considered to be of interest.

Stakeholders directly informed our communication plan. They recommended results be disaggregated by drinker types and wealth groups primarily, with sex and age group impacts of secondary importance. They suggested focusing on presenting graphs and tables, quoting figures where needed.

In relation to whom to communicate with and how, participants indicated that policy makers were the top priority for communication, but also the media, civil society organisations and the public. A one-page policy brief was the most commonly mentioned form of communication, complemented by online articles with infographics, presented in person, and online interactive web tools if possible. Following circulation of the outputs document, stakeholders confirmed we had captured the results of the workshop and suggested a number of people to invite to the final workshop.

3.4 Outcomes of Phase Four: Communicating Research Evidence to End-Users

The three preselected discussants and the attendees provided valuable information, different perspectives, criticisms, areas of concern, and areas of confusion. Topics discussed included differential alcohol prices, impacts on ‘shebeens’ (largely smaller unlicensed alcohol outlets found in informal settlements), and communication materials. Stakeholders suggested ways we could improve our communication of results, namely presenting percentage changes as well as absolute changes in consumption, and explaining the increase in spend results from increased prices despite purchasing less. Participants requested permission to share our slides with their networks, increasing research communication.

Creating positive working relationships with the stakeholders led to growth in attendance at each phase with 12 initial interviewees, then 12, 16 and 29 stakeholders attending each subsequent event. This provided opportunities for communicating the research beyond the final workshop and academic papers. We received invitations to national conferences, a live radio interview, further formal research collaborations developing the model to a local context, and presenting evidence to ministers. The research also resulted in two academic publications.

4 Discussion

In this paper we have demonstrated the feasibility and value of stakeholder engagement in the development and communication of a public health economic model in South Africa. We found there to be many benefits of taking this approach, which included increasing our initial understanding of the policy context, establishing key stakeholder relationships, and providing specific guidance on data, assumptions and important outputs, which allowed the model to be highly tailored.

There were a number of strengths of this work. Starting with one-to-one interviews, which created a strong sense of connection that promoted attendance at the next phase of engagement. The workshops used for co-producing model focus and structure and scrutinising model development were highly interactive, providing written outputs for analysis following the sessions, helping to inform certain modelling decisions (such as choice of certain outcomes and data sources). The interactivity also maintained engagement and energy, according to stakeholder feedback. We created a strong feedback loop via the output documents and revisiting, at the beginning of each session, how stakeholder input had directly informed the work. Attendance grew at every subsequent phase as stakeholders not only stayed with the research project but also invited colleagues. At the end of phase two (co-producing model focus and structure), there was a request for increased policy engagement. By phase three (scrutinising model development and communication planning) it was noted that government representation had grown, particularly across different departments. The switch to an online workshop format, initially very challenging, was a strength in engaging more policy professionals who may have been reluctant to sacrifice the time if travel had been involved.

We acknowledge a number of limitations. Moving the engagement online, although helpful for many, reduced the input of stakeholders with poorer access to technology. These were exclusively stakeholders from civil society organisations. The online format advantaged those who work in government or universities with access to technology; this is particularly pertinent in a low- and middle-income country (LMIC) setting. A further limitation was that we did not engage stakeholders from every potential category, such as alcohol treatment practitioners, the general public, police or the alcohol industry. We acknowledge that our selection of stakeholders focused on those we might expect to be pro alcohol-pricing policies and avoided those we would expect to be potentially hostile, such as the alcohol industry. This decision will have influenced the way in which the model was developed, for example we chose to focus time on modelling the impact to health as opposed to different market and supply chain responses. This could have an impact on the acceptability of the findings to excluded groups, such as the alcohol producers, and diminishes our ability to assess the potential scale of changes in the supply chain that may mitigate the impacts of MUP.

There was an ongoing challenge between managing timelines of the engagement, to maintain interest and provide final results, with the slower academic process of receiving peer review of publications that might mean the final results require revision. Our pragmatic solution was to provide work with caveats, following up when final results were published in academic journals. Finally, we have not formally evaluated the engagement process and are yet to observe the implementation of the policy, which will allow us to assess the role that our research, and in particular the stakeholder engagement, played. However, we are engaging in ongoing collaborative research with a provincial government, facilitated through our relationships with governmental stakeholders established during this research project.

It is broadly recognised that stakeholder engagement is increasingly being used, and health economic modelling guidance states the need for this to be clearly incorporated in the reporting of economic evaluations [6, 23, 24]. However, we found little practical advice in precisely how a modeller might engage stakeholders in public health modelling, although some examples exist relating to health technology assessment [25].

Despite the heterogeneity of public health economic modelling, based on our experience, we consider the following stages and associated activities to be generalisable for those interested in meaningfully embedding their research in context and making it useful to decision makers. Firstly, developing a detailed understanding of the local policy context via listening to local expert stakeholders (not only academics) before conceptualising the problem or the model. Next, co-producing model focus and structure and then allowing for scrutiny as the model is developed. Finally, there is a responsibility to communicate the results of the modelling transparently to all those stakeholders who have invested their time in the work as well as interested parties beyond the stakeholder group.

There are several issues that have not been resolved adequately by the research to date, including ours, such as how to ensure inclusivity, how to be explicit about values and beliefs, and how these influence all modelling and stakeholder engagement decisions, how to manage disparate interests, and how to manage the convergence of a short policy window with a slow academic process. Research specifically in relation to these aspects of stakeholder engagement would be valuable, as would learning from those who have applied a general approach to developing integrated and tailored stakeholder engagement.

5 Conclusion

As a modelling team we invested time in listening to local stakeholders throughout the modelling process, viewing it not as a parallel work stream but as a fully integrated component of the research. We realised many benefits including increased contextual understanding, the establishment of ongoing working relationships, a highly tailored model, and opportunities to communicate our research widely.