How Does Evidence Matter? Understanding ‘What Works’ for Wellbeing
- 781 Downloads
In 2015 the UK launched an independent What Works Centre for Wellbeing, co-funded by government departments and various agencies, which aims to develop a ‘strong and credible evidence base’ to help promote wellbeing in policy. Yet while there is widespread agreement that evidence matters in policy-making, it is far from clear what kinds of evidence matters, in what circumstances and to what extent. In this context, this article presents the findings of research exploring with policy-makers and stakeholders issues in the use of evidence in relation to wellbeing in public policy. In particular, it highlights evidence as a specific form of (research-based) knowledge and considers the importance of this relative to other forms of knowledge; political, professional and experiential. This approach highlights a broader understanding of ‘what works’ beyond the relatively technical sense often employed to describe the work of What Works Centres and in the use of evidence more generally.
KeywordsWellbeing What works Evidence Public policy Politics
Wellbeing is increasingly important in politics and policy across a range of contexts and not least in the UK. In 2015 the UK launched an independent What Works Centre for Wellbeing (WWCW), co-funded by government departments and various agencies, which aims to develop a ‘strong and credible evidence base’ to help promote wellbeing in policy. In this context, this article presents the findings of research exploring with policy-makers and stakeholders on the use of evidence in relation to wellbeing in public policy. In particular, it highlights as evidence as a specific form of (research-based) knowledge and considers the importance of this relative to other forms of knowledge; political, professional and experiential. This approach highlights a broader understanding of ‘what works’ beyond the relatively technical sense often employed to describe the work of What Works Centres and in the use of evidence more generally.
This paper has six further sections. The next section charts the rise of wellbeing in policy in the UK leading to the creation of the WWCW. Section three considers the use of evidence in policy, highlighting both practical developments in the UK policy arena and important academic contributions to understanding this phenomenon. Section four provides an overview of the research design for this study, while section five presents the main findings. These findings are analysed in the sixth section, referring back to the analytical themes outlined in section two. The concluding section reflects on the key findings and sets out the main contribution of the article.
2 The Rise of Wellbeing in Policy
The ONS subsequently conducted a public consultation on what domains and measures of wellbeing should be used for the programme. In response to this consultation, ten domains were decided upon in July 2012: individual wellbeing (later renamed ‘personal wellbeing’1); our relationships; health; what we do; where we live; personal finance; economy; education and skills; governance; and natural environment. Most of the data for these domains was already being collected by the ONS, and was simply repackaged for the wellbeing framework. However, four new subjective wellbeing questions were added to provide data for the individual wellbeing domain. The four questions were: Overall, how satisfied are you with your life nowadays? Overall, how happy did you feel yesterday? Overall, how anxious did you feel yesterday? Overall, to what extent do you feel the things you do in your life are worthwhile? Each question is measured on a scale from 0 to 10. Data for the programme has since been collected annually. Subjective wellbeing questions were new to national statistics and it was this part of the programme that attracted most attention.
…open up a national debate about what really matters, not just in government but amongst people who influence our lives: in the media; in business; the people who develop the products we use, who build the towns we live in, who shape the culture we enjoy. And… this information will help government work out, with evidence, the best ways of trying to help to improve people’s wellbeing (Cameron 2010).
While there had been interest in wellbeing in UK government circles before 2010 (see Bache and Reardon 2016, 71–72), it was the launch of the ONS programme and Cameron’s support for the idea that firmly placed it on the government agenda. In 2011, the Treasury updated its guidance to government departments to include subjective wellbeing in policy appraisal, alongside the long-established market-based approaches of Stated Preference and Revealed Preference (Fujiwara and Campbell 2011, 57–58). However, the government was clear that the ONS subjective wellbeing indicators were still in development and that ‘we should not expect at this stage to have examples of major decisions that have been heavily influenced by wellbeing research’ (HM Government 2013, para. 4). However, it also suggested that it had put in place some ‘new foundations’ for instilling a wellbeing approach (HM Government 2013, para. 4). Among these new policy foundations were a number of new departmental surveys and narratives relating to wellbeing and examples of wellbeing emerging in policy appraisal and evaluations (on these developments see Bache and Reardon 2016, 72–80).
In October 2014 the government announced that funding would be provided for the creation of an independent What Works Centre for Wellbeing (WWCW). The Centre, which has 17 founding partners, including government departments and the Economic and Social Research Council (ESRC), aims to develop a ‘strong and credible evidence base’ which will support organisations ‘to concentrate efforts on interventions that will have the biggest impact’ on wellbeing (Cabinet Office 2015). Funding for the Centre was confirmed after the 2015 general election, which gave a strong signal of the government’s continuing commitment on the issue—and continuing faith in the value of evidence in policy. The creation of the WWCW followed the establishment of What Works Centres in a number of other policy areas: crime reduction, health and social care, education, early intervention, ageing better and local economic growth. The centres aim to ‘help to ensure that policy makers, practitioners and commissioners can make informed decisions based on impact and cost effectiveness’ (Cabinet Office 2015).
In comparison with other policy areas covered by What Works Centres, wellbeing is more recent to the policy agenda and is arguably subject to greater contestation over definition and measurement. An evaluation of the use of wellbeing powers in UK local government found different interpretations and discourses of wellbeing across local authorities and ‘little consistency’ in understanding of the new powers: ‘there was indeed no ‘single’ problem of wellbeing but rather a range of ongoing problematisations, which varied according to the interpretations of different stakeholders’ (Griggs and Howarth 2011, 221). Such conclusions are reflected in the wider literature on wellbeing (e.g. Scott 2012; McGregor 2015) and conflict over the nature of the problem and thus the potential solutions have led to it being described as a quintessential ‘wicked problem’ (Bache et al. 2016).
This definition set out common ground for the different programmes in the WWCW but, as the ESRC acknowledged, would not resolve debate over definitional issues. The extent to which personal/subjective wellbeing should be emphasised remains a particularly contentious issue (below).
Wellbeing, put simply, is about ‘how we are doing’ as individuals, communities and as a nation and how sustainable this is for the future. We define wellbeing as having 10 broad dimensions which have been shown to matter most to people in the UK as identified through a national debate…[see above]… Personal wellbeing is a particularly important dimension which we define as how satisfied we are with our lives, our sense that what we do in life is worthwhile, our day to day emotional experiences (happiness and anxiety) and our wider mental wellbeing.
The newness of wellbeing on to the policy agenda means that much extant research has not been conducted with this definition in mind and, often, relevant research has wellbeing as a secondary or even tertiary outcome (ESRC 2014, 5). As such, part of the task of the WWCW is to address challenges such as ‘as the ability to establish cause and effect from such evidence’ (ESRC 2014, 5). In short, the accumulation and transmission of evidence on how policies can enhance wellbeing is seen as a crucial next step in bringing wellbeing more fully into policy.
3 The Use of Evidence in Policy
The Labour Party’s 1997 manifesto statement ‘what counts is what works’ (Labour Party 1997, 2) was a signal of a post-ideological approach to policy that it sought to bring into government after its election in 1997. The shift towards evidence-based policy occurred in other places around the same time, linked to a broad shift in managerial reforms focusing on effectiveness and efficiency (Marston and Watts 2003, 147). In the UK context, one consequence of this shift was the creation of ‘What Works Centres’, beginning with the National Institute for Health and Care Excellence (NICE) in 1999. According to the ESRC (2017):
This [White Paper] recognised the need for policy making to be more responsive to citizens’ demands; looking forward; evidence-based; properly evaluated and based on “best practice”(GSRU 2007, 9).
What Works centres enable leading social scientists to evaluate the availability and quality of evidence underpinning public policy interventions, compare the effectiveness of interventions, and advise those commissioning and undertaking interventions to ensure that their work can be evaluated effectively. They also identify research and capability gaps, working with partners to fill them.
Two illustrations of simplified hierarchies of evidence based on study design.
Reproduced with permission from Nutley et al. (2013, 10)
Level I: Well conducted, suitably powered randomised control trial (RCT)
Level II: Well conducted, but small and under-powered RCT
Level III: Non-randomised observational studies
Level IV: Non-randomised study with historical controls
Level V: Case series without controls. Source: Bagshaw and Bellomo (2008, p. 2. 7)
Systematic reviews and meta-analyses
RCTs with definitive results
RCTs with non-definitive results
Case control studies
Source: Petticrew and Roberts (2003, p. 52)
A number of reasons are advanced for using evidence, from helping to understand an issue, to identifying the appropriate policy response and shaping future thinking (Nutley et al. 2013, 10). Thus, it is used at various stages of the policy cycle, from creating, developing and implementing policy to defending and justifying a policy decision.
While it was common in this early period to refer to ‘evidence-based’ policy, growing recognition of the many drivers of policy making (e.g., GSRU 2007; House of Commons Science and Technology Committee 2006; see also below) shifted the discourse more towards evidence-informed policy to acknowledge that policy is rarely a simple and direct response to evidence. Head (2010, 80) developed a more comprehensive account of why evidence ‘can inform policy rather than constitute a systematic foundation for the policy process’. First, that a strong research base is simply not available to policy makers in some areas; second, decision-makers are often motivated or influenced more by factors other than research evidence; third, that where evidence is available it is often poor; fourth, professional knowledge rather than evidence-based is often more important; and fifth, evidence-based policy appears to have less influence in areas that are in flux (Head 2010, 80–81).
While research-based evidence can provide an appearance of scientific neutrality that holds a particular status and authority in policy-making, its influence is shaped by the nature of the policy context into which it enters. As suggested above, evidence is generally not the only contender for influence and politics and knowledge of different types interact to shape decision-making.
In the second and third categories, our questions are changing as well as our answers. In such situations, evidence does not exist in the abstract, floating free. It exists in relation to theories and concepts that provide the prisms through which the world is seen. These theories are not alternatives to hard facts and evidence: they are the only ways of making sense of them (Mulgan 2005, 222).
Weiss’s (1979) seminal contribution on research utilisation contrasted political uses of research with decisions directly driven by evidence, suggesting that the latter were particularly rare in relation to social science research. In a later paper she explained further this political dimension, suggesting that ‘When new data or research findings arrive on the conference table in the councils of action they confront four I’s already sitting at the table…ideology, interests, institutional norms and practices, and prior information’ (Weiss 2001, 286).
Other scholars have sought to refine the political dimension further. Boswell (2008) highlighted the symbolic functions of knowledge: one being a legitimizing function, in which an organisation uses knowledge to enhance its legitimacy in a particular policy area; the other a substantiating function, in which knowledge lends authority to help substantiate organisational preferences where there is political contestation.
An inappropriate choice of data, their placement at a wrong point in the argument, a style of presentation that is unsuitable for the audience to which the argument is directed – any one of these factors can destroy the effectiveness of information as evidence, regardless of its intrinsic cognitive content… the applicability of evidence depends on a number of features peculiar to a given situation, such as the nature of the case, the type of audience, the prevailing “rules of evidence,” and even the persuasiveness of the analyst(Majone 1989, 48).
In such cases, the way a problem is framed is important both for the scientific validity and for the political management of complex problems, which ‘are unlikely to be “solvable” through a single policy instrument or “magic bullet”'(Head 2010, 83). It is for these reasons that wellbeing has been described as a ‘wicked problem’ (above).
In relatively simple issues where all the variables can be specified and controlled, methodological rigour is likely to be tight, with some confidence that causal factors can be clarified. But in programs with multiple objectives, or where the clients/stakeholders are subjected to many sources of influence beyond the scope of the program, the challenge of accurate understanding is compounded (Head 2010, 83).
John Shepherd’s (2014) report to government on the role of What Works Centres,2 which pre-dated the creation of the WWCW, provides a useful bridge between these analytical themes and examination of the case of wellbeing. Shepherd’s starting point is that ‘the creation and adoption of effective policies, programmes and interventions depends on a functional evidence ecosystem’ (Shepherd 2014, 5). He suggests that: ‘What Works Centres are an essential part of this ecosystem and need to be concerned not just with evidence synthesis and adoption but with the whole system in their sector so that faults can be identified and put right’ (Shepherd 2014, 5). Shepherd’s approach focused on four features of the evidence ecosystem: evidence sources; transmission lines; problems; and incentives. This conceptualisation provides a structure through which to examine the themes highlighted above relating to the factors that constrain or facilitate the use of evidence and the influence of other factors on this complex issue; in particular, the role of other forms of knowledge.
4 Research Design
- 1.Key terms and issues
Wellbeing—how do you understand this term? How do they think other people understand it?
‘What works’—what does this mean to you?
- 2.The evidence ecosystem
Evidence sources—what sources of evidence do you know about and what are the ones you use?
Transmission lines—what are the channels through which your organisation receives evidence?
Incentives—What are the main incentives for using evidence?
Problems—What are the main challenges around the use of evidence?
- 3.The challenge of wellbeing
Does the issue of wellbeing present specific challenges in the use of evidence? If so, of what types?
Other than research evidence what other forms of knowledge are important to wellbeing (e.g., political, professional, experiential)?
- 4.Moving forward
What specifically do you want from evidence providers/the WWCW?
The target population consisted of individuals with a close involvement with wellbeing in their work and the sample chosen to reflect a cross-section of this estimated population of interest6: one that sought to include all significant elements. The selection of organisations was undertaken jointly with stakeholders from the WWCW who are close to the embryonic wellbeing policy networks and who were thus able to advise on relevant participants for this study. Through these connections I was able to gain access to senior actors who might otherwise have not been available for interview. While it was not the main aim of the research to identify differences in approach or emphasis across organisations, some differences did emerge and are reported below.7
Interviews were the preferred choice of data collection in order to provide a deep understanding of the phenomenon of evidence use in complex policy arenas and to provide insight into the meanings of the subject’s experiences. The use of semi-structured interviews ensured that the main themes and questions were addressed in all interviews but also treated the interviewees as ‘active subjects’ able to organise their responses within their own frameworks (Halperin and Heath 2012, 299).
In the first stage of analysing the findings, the interview transcripts were read holistically, looking at patterns (similarities and differences) in the data to facilitate the identification of important themes beyond those that provided the guide to the interview questions. The most important and interesting parts of the text were highlighted to provide a coding frame: all material was coded for relevant themes. The sections of text marked with each code were drawn together and conclusions drawn by analysing the meanings of the data and their implications for the research questions. The data from different respondents was cross checked to validate key findings. To provide verification, the findings were presented to a workshop of WWWC stakeholders from a range of organisations and other policymakers, some of whom were interviewed for the research.8
5 Research Findings
5.1 Understanding of Key Terms and Issues
culturally, some commentators across the UK view wellbeing as something that’s attributed to and in control by individuals… sometimes the media characterise wellbeing as happiness, which is extremely unhelpful… (interviewee #5).
The findings illustrated the complex and contested of nature of wellbeing and the co-existence of both broad and narrow definitions in the policy process. The conflation of wellbeing and happiness is quite common in the UK, not least because of the novelty of subjective wellbeing within the ONS framework and the emphasis often placed on it in policy circles (Austin 2016). A common theme of the interviews was that these definitional issues remain important for understanding the role of evidence in policy and these are discussed further below.
Professionals in different areas will give different definitions - emotional wellbeing, mental wellbeing… You ask ten people, you get 11 different answers, essentially (interviewee #6)
Identifying, based on evidence, what works and then doing something with that information so it leads to change (interviewee #7).
…proven interventions and approaches (interviewee #6).
In short, most responses reflected a technical notion of what works, although it was clear from answers to other questions that interviewees were also very clear of the importance of a range of other factors influencing policy (below).
5.2 The Evidence Ecosystem
5.2.1 Evidence Sources Used
Evidence sources used (in no particular order)
Commissioned research and evaluations
Randomised control trials
Systemic/meta-reviews (e.g., Kings Fund/NICE/New Economics Foundation)
Voluntary and community sector studies
Syntheses of academic literature (e.g., by think tanks, voluntary and community organisations)
All-Party Parliamentary Groups
Government reports (e.g., Cabinet Office, Department for Work and Pensions)
Government surveys (e.g., ONS, Cabinet Office)
I think it’s important that the ecosystem of evidence is equally valued and equally mined but that when one is looking for some direct correlation between an input and an output that might be subject to slightly more rigorous scientific, methodological means (interviewee #5).
There was a clear sense across interviewees that government preferred particular types of evidence that tended to reflect the hierarchies discussed above. However, for other organisations, particularly within the voluntary and community sector and within some local councils, participatory approaches involving their communities and/or stakeholders provided an important source of legitimacy. In some cases, these types of approaches—such as co-production—were privileged over research-based evidence because of the greater legitimacy they were seen to bring to organisations and their chosen course of action.
5.2.2 Transmission Lines
Transmission lines (in no particular order)
In-house research and light touch evaluation, literature reviews, evidence-gathering
Wider sector networks
Policy advisory groups
Guidance from national bodies (e.g., Public Health England)
Circulars (NICE, National Health Service, Kings Fund etc.) around specific issues, such as obesity.
Search engines (e.g., Pub Med)
Conferences (‘useful for finding out what’s happening in other organisations’)
‘People send us things’ (e.g. emails from think tanks)
Meetings (e.g., with Alliance for Useful Evidence, NESTA)
Information services (i.e., regular bulletins)
Private consultancy organisations
The range of transmission lines indicates something of the breadth and complexity of the idea of wellbeing. Bodies such as NICE and Public Health England were clearly more important to those organisations with a more medicalised conception of wellbeing, while the Legatum Institute’s research (O’Donnell et al. 2014) appealed more to organisations that focused on subjective wellbeing.
Generally, interviewees emphasised the importance of professional networks and personal contacts: ‘You have the right conversations to make sure you’re not missing any tricks’ (interviewee #9). Being well connected in this way meant that individuals often did not have to seek out evidence but found it came to them. However, it was acknowledged that this could skew the evidence received. Such comments illustrate the precariousness of assumptions that policy-making is based on systematic consideration of the best evidence available.
Civil servants also identified the importance of evidence in justifying policies to the public, which would be a consideration for politicians in particular.
If you’re publishing a policy you’ve got to back it with evidence… [There are] decision-making gates to get through for big policies, like impact assessments and business cases, spending reviews. So big incentives are built into the system in some respects… (interviewee #6)
Incentives for the use of evidence (in no particular order)
Quality of ‘own’ work
Value for money
Producing better policies
To get more funding
To learn and improve
Appraisal processes within Whitehall
Generally, interviewees from all organisations seeking to influence government acknowledged that sound evidence is essential. As one local government interviewee stated: ‘We want this evidence to be so strong that the government and the Treasury cannot turn away the findings because they’re methodologically unsound’ (interviewee #9). Concern with demonstrating value for money, which had intensified in the context of economic recession, was a key factor both within all individual organisations and in the wider policy arena.
it’s even more important in the current climate in that there is a definite view that charities are kind of ideologically-driven, political mouthpieces… you leave yourself very, very vulnerable to [this criticism] if you’re not evidence-based (interviewee #7).
Problems with the use of evidence
Awareness (‘knowing that it exists’)
Understanding who the evidence is for and why they need it
Lack of staff skill in using evidence
Timeframes (i.e., within which impact has to be seen)
Timeliness (i.e., evidence not available when it is needed; legislative cycle not right)
Time pressures (on staff)
Access to evidence (e.g., academic journals)
Policy making is ‘messy’ (not rational and linear)
Lack of clarity about where the evidence is coming from
Academic work not accessible/practical (‘think tanks put recommendations on first page’)
Lack of clarity on the relative strengths of the evidence
Challenging the default position
Evidence focused on individual outcomes not social (e.g., ‘improving blood pressure, not social capital’)
Qualitative evidence less valued
Sheer volume of research
Evidence contradicts the principle of co-production
The quality of evidence was an important concern across organisations, with one interviewee stating ‘I think there’s a lot of really bad evidence that gives wellbeing a bad name out there… a woolly kind of perspective’. There was also common concern that the relative strength of the evidence available was often insufficiently clear. As one interviewee suggested:
It can take a generation to influence a community… but it is difficult to plan in advance with financial uncertainty. It stifles innovation and planning. You look for options that can give more immediate results (interviewee #2)
I think it is beholden in publications to say, “Look, this is what we’re basing these findings or statements on and here’s the levels of evidence”. So it’s just that much more honest and open, transparent way of which evidence is being used. And letting people make up their own mind to a certain extent (interviewee #5)
Interviewees across the board pointed to the importance of competing issues and interests, political commitments, public opinion and other forms of knowledge (see below). Such comments highlighted a general awareness of the need to understand ‘what works’ beyond the technical sense of the term.
…it’s not [the case of] here’s a problem, let’s explore the problem, let’s engage the public in a problem, let’s look at the evidence, let’s commission new research around it, and have lots of time to do all of this and make the best use of evidence that you can. Then come up with solutions, appraise the options, do all of that type of stuff. I don’t think that happens very often (interviewee #6).
5.3 Other Forms of Knowledge
Awareness of the importance of political knowledge was most prominent in the highly politicised environments of local and central government. One local government official highlighted the importance of political knowledge in the following way:
People understand that the evidence ecosystem is pooled and shaped and manipulated in different directions by each of those different interests, whether it’s local advocacy, whether it’s political advocacy, whether it’s economic advocacy (interviewee #5).
Another local government official (interviewee #10) made this point even more forcefully: ‘the political decision has much, much more power than everybody else. So, you know, I can do all this work and then if the political people don’t like it, they can just stop it’. She also highlighted the challenge that wellbeing faced in such a politicised context, suggesting that to privilege (subjective) wellbeing measures above other kinds would mean a change in the ‘entire culture of the council’. This point was echoed by a national civil servant:
… about a third of it you go in with evidence but you’ve also got the political knowledge of - especially talking about councillors - what are their political parties? What wards are they representing? Particular issues in their ward? What sort of things they’ve funded or supported before? It’s putting it through that template of politics with a small ‘p’ at the local level and all [of] your understanding [of] the structures that the council operates within… (interviewee #2).
Experiential knowledge emerged as an important theme across organisations but was emphasised most by voluntary sector interviewees. Comments included:
It’s very difficult for politicians to talk about wellbeing because people feel that politically there’s not demand to express things in that form. They will focus on, for the most part, much more concrete intermediate goals, like you know, how can you reduce unemployment? They tend not to think, fundamentally in terms of wellbeing, at the moment, and particularly in politics, people are afraid that they will be ridiculed for doing so (interviewee #4).
Experiential knowledge is really important… knowing the local market is increasingly important (interviewee #7 – voluntary sector)
Service users… can throw up implementation issues (interview #11 – charitable organisation)
One local authority official explained how her council emphasised the co-production of decisions with local residents and stakeholders: this was central to its philosophy as a cooperative council. This approach presented tensions with an evidence-based approach:
We proceed on a case-by-case basis… stakeholder experience of the problem, frontline workers who are engaged in your policy or who could help to deliver solutions (interviewee #6 – national civil servant)
One charitable organisation also emphasised participatory approaches to shaping policies more relevant to local needs. One project led to the development of an index to guide policy based on what local people saw as the most important things in their lives. Such approaches bring potential advantages in terms or organisational and policy legitimacy but the interviewee suggested that this project ‘probably has some gaps methodologically’ that would make this approach less attractive in other contexts.
One of the big problems that I face, at the moment, is that it [the idea of evidence-based policy] is constantly used by people saying to me or my colleagues [that] as long as it’s got an evidence base then it’s bound to work. And that is very, very frustrating because that’s not the case and therefore what it means is that it contradicts [the principle of] co-production (interviewee #10)
There was consensus among interviewees that much policy-making depends heavily on professional knowledge. As noted above, evidence may not be available or, at least, available at the right time. In relation to a decision over whether to invest in a significant spending programme, one NDPB official stated: ‘we’ve taken a punt on what we understand… we know that the evidence to prove that is limited. If I waited for the evidence base to prove it I wouldn’t have taken the punt’ (interviewee #9). Individuals representing membership organisations highlighted the value of professional knowledge supplied by their constituent organisations, who were often consulted on key issues.
5.4 The Challenge of Wellbeing
There were differences between those who believe a broad range of indicators are needed and others who think that subjective wellbeing indicators are a way forward, pragmatically at least. On this issue there was no consistent pattern across organisations. So, on the one hand:
It does need to move us into areas of scientific inquiries that are much more integral or integrated across disciplines. And that challenges the way in which academia currently organises itself and scientific funding is currently distributed (interviewee #5).
And, on the other:
They say we can’t reduce wellbeing down to a couple of questions. No, you can’t, but if you think that it’s practical always putting 20 or 30 wellbeing questions into a survey - it’s very naïve to think that’s the case. You have to sacrifice yourself and basically, reduce or boil down to a few questions (interviewee #6).
However, there was also a common view across organisations that contestation over definition and measurement were to some extent inevitable and that, as one interviewee put it, ‘There’s no point in criticising anyone’s approach on wellbeing because it has to be administration/context-specific’ (interviewee #11).
the use of subjective wellbeing as a dominant measure is because we – a lot of people - are looking for simplistic answers to very complex questions… we need to balance that with much greater objectivity (interviewee #5).
5.5 What Policy-Makers Want from the WWCW
What policy makers want from evidence providers
Provide information that is simple and short
Prioritise thinking (i.e., provide a specific focus within wellbeing)
Develop a strong brand for the WWCW (so that it can be trusted)
Be collaborative (not least with ‘people on the ground’)
Improve accessibility of evidence
Improve availability of evidence
Make sure evidence is relevant
Promote different types of evidence sources
Link with other What Works Centres (because of the multidimensional challenge)
‘Share information and problem-solve about things that are happening’
Scale up examples of good practice (e.g., within a community)
Provide specialist knowledge and advice
Build the capacity of organisations and people to do their own analysis and generate their own wellbeing evidence(particularly the voluntary and community sector)
Address definitional issues and help to build consensus around the term ‘wellbeing’ and related metrics (e.g., in government)
Find more innovative, creative and successful channels of evidence transmission’
Give evidence beyond what might be politically expedient at the time
Create a demand for wellbeing information, evidence, knowledge, science, and statistics.
Translate new law and policy (e.g., explain the political and legal levers and how that translates at the local level’.)
Support small scale organisations/voluntary and community sector
Provide an online resource bank for different methods/latest research
Have direct contact with organisations
Provide clarity around legitimacy given to different ideas of looking at wellbeing (e.g., ‘subjective wellbeing is only one way’)
Provide insight into upcoming/emerging trends on wellbeing
Give ‘concrete examples’ about things that worked successfully
Explain why something worked
Provide examples of successful co-production
A common theme was the need for complex research to be translated into short and accessible information. A number of interviewees identified a rapid growth in research on wellbeing—an ‘exponential curve’—but suggested that it has ‘tended to be a very academic subject’ that evidence providers need to ‘convert into something that’s very clearly actionable and can influence decisions’ (interviewee #6).
A number of interviewees also raised the issue of whether the WWCW should have a specific focus (e.g., mental wellbeing, subjective wellbeing). One gave the example of how the Joseph Rowntree Foundation had focused specifically on poverty and had developed a respected reputation on this issue. However, there was no clear consensus among interviewees on what this focus should be for the WWCW
Another interviewee suggested that ‘sometimes evidence gets used because it’s the most visible evidence' and the Centre should ‘bring a lot more evidence into play to make that more visible’ (interviewee #5).
…a much more modern way of addressing the issues; a much more integrated sense of all of these things together. And that lends itself to, I think, new and potentially very exciting forms of evidence… the What Works Centre for Wellbeing needs to promulgate that ecosystem and multiplicity of what we consider as justifiable evidence (interviewee #5).
Finally, some interviewees believed the Centre might also play a key role over definitional issues and in promoting a ‘common currency’ for comparison of wellbeing interventions. However, while the ONS subjective wellbeing questions were seen as having the widest recognition in this regard, and for some were the best way forward (above), there was no consensus on this issue: other interviewees suggested that the WWCW should focus on setting out different measures for different purposes.
There was consensus among interviewees that the accumulation of evidence is an important next step for wellbeing in policy. The interviewees indicated considerable demand for evidence of different types and for a range of purposes. While there are issues about definition and measurement outstanding, and an ongoing challenge to persuade key actors that this is an important policy issue, interviewees generally believed that scepticism around wellbeing had receded significantly in recent years. As one put it: ‘I think people do understand that wellbeing is important… They don’t think it’s mad anymore to be measuring this. I think what they want is to do something about it’ (interviewee #6). Central to this purpose is greater understanding of what policies can enhance wellbeing, for whom and by how much. This is where evidence can play an important role.
In Mulgan’s (2005) terms, wellbeing can be considered an inherently novel policy field. As such, less is known about ‘what works’ in policy than in more established fields. The networks are less well established and there are fewer ‘trusted’ brands’ in terms of evidence providers. The creation of the WWCW is aimed at addressing these issues. However, as wellbeing moves from an abstract idea to one that is being defined more precisely for use in policy it also demonstrates characteristics of a policy in flux. What counts as relevant evidence is strongly contested and there are disputes over the basic theoretical approaches. Should, for example, wellbeing be conceived as individual wellbeing measured by subjective wellbeing indicators? Or should wellbeing be conceived as a multidimensional phenomenon in which subjective wellbeing indicators are placed alongside a raft of other, more objective indicators? The answer at this stage is not clear: while the official statistics of the ONS combine 41 subjective and objective indicators across ten domains, there is also a push to privilege subjective wellbeing, which is controversial in some quarters. As noted above, there is no consistent pattern of views across or within organisations on this issue. The interview responses on this issue were informed more by the particular views of individuals, shaped by their different theoretical perspectives and values.
Yet the WWCW has been encouraged to pay particular attention to subjective wellbeing: as a potential ‘common currency’ that would allow the effects on wellbeing of different policy interventions to be compared against the same standard more easily. This may have the effect of privileging certain types of evidence over others and may also discourage some organisations (and/or individuals) from engaging with the issue if this approach is not seen as legitimate. However, while this approach has been encouraged for the purposes of the WWCW, this is by no means the end of the matter. Indeed, while the ESRC identified personal wellbeing as a ‘particularly important dimension’, it also acknowledged that the ONS framework is broad and that some aspects of this framework would be emphasised more in some policy areas than others; and, that other dimensions may also emerge (ESRC 2014, 4). In short, as Mulgan’s arguments would suggest, in the field of wellbeing the questions are changing as well as the answers and evidence—far from floating free—is being generated and accumulated in relation to different theories and concepts that frame wellbeing in different ways. As such, these theoretical issues are inseparable from the evidence and are the only way of making sense of it.
As Majone notes, there is no easy separation of facts and values and, while much of the language of ‘what works’ implies a rational, technical perspective, how evidence is communicated is crucial. This is not simply about clarity and sound methodology but also about understanding the audience and the context. A common theme emerging from the interviews was that evidence serves very different purposes in different contexts and this shapes the nature of the evidence sought. Thus, civil servants and others seeking to demonstrate efficiency and value for money may be drawn more towards quantitative evidence, while organisations seeking different types of legitimacy (see Boswell above) may prefer participatory approaches.
Such findings highlight the usefulness of Head’s distinction between different types of knowledge. Evidence-based knowledge is an important form, holding a particular status through its perceived neutrality; although, as the research here suggests, values and ideology are not only present in the demand-side world of politics but also in the epistemic communities that supply the evidence. As noted above, researchers adopt different theoretical perspectives on wellbeing that are informed by different value structures. These perspectives determine the ‘facts’ and the data collection methods that are seen as relevant. However, explicit consideration of political, professional and experiential forms of knowledge provides nuance to understanding the role of evidence beyond the simple rational versus political dichotomy.
Political knowledge draws attention to a range of factors—values, ideologies, manifesto commitments, electoral competition, party factions, interest group pressure, the media, and public opinion, to name just some—that combine in the minds of political actors to provide a sense of what is politically feasible and what is politically desirable. So, on the one hand, the pursuit of wellbeing might be considered desirable, but not feasible for any of these reasons. On the other hand, it is plausible at some future point that the pursuit of wellbeing through policy may become politically feasible but is seen as undesirable because it conflicts with other goals deemed more desirable. These issues are more pronounced in the more politicised arenas of central and local government.
Professional knowledge is arguably dominant in much routine policy-making. While some public policy decisions are widely debated and various actors have a voice in the process, much policy making takes place in the relatively closed confines of well-established policy communities. Policy communities tend to have embedded norms and practices that constrain the potential for new policy ideas to take hold. Policy specialists know best ‘what works’ in and for their respective communities and this tends to provide a bias against abrupt shifts in direction. This does not mean that new ideas and new evidence cannot lead to significant change, but that they meet with an inbuilt tendency towards inertia when they arrive fresh at the table: a theme that emerged across organisations.
Experiential knowledge draws attention to the importance of end users, communities and residents’ ideas about ‘what works’ for them. This approach to ‘what works’ eschews the search for a ‘one size fits all’ approach to defining, measuring and delivering policy for wellbeing. It may mean a trade-off between generating evidence through means conventionally viewed as more robust methodologically in favour of capturing the views of the specific target audience for policy. As noted above, such an approach offers legitimacy for the delivery organisation and, in theory, for the specific policy that emerges. It is an approach that has acknowledged limitations in terms of scalability, and is thus less attractive to large governmental units than smaller community-based organisations, but is one that nonetheless provides an alternative to established assumptions of what constitutes valid knowledge.
Wellbeing in policy has attracted increasing interest in a range of arenas. In the UK, developments have reached the stage where the government has created a What Works Centre to bring together the available evidence to support those seeking to bring wellbeing more fully into policy. This is seen as a crucial ‘next step’. However, as is widely recognised and as this research has illustrated, evidence is just one factor among many in shaping policy developments: and often not the most important. In this context, this article has sought to shed light on the role of evidence in promoting wellbeing in policy. What the research shows is an issue that is complex, relatively new to the policy agenda and one that has contestation at its heart. In such a context, understanding ‘what works’ is a major challenge.
There is widespread agreement that evidence matters, but it is far from clear what evidence matters, why and to what extent. To begin to form answers to these questions requires a broad conception of ‘what works’ that treats evidence as one form of knowledge that interacts with other forms to shape policy. No-one believes there is a ‘magic bullet’ that can resolve deeply held differences over the nature of wellbeing, how it should be defined and measured, and the extent to which governments should seek to promote it through policy. However, this article does highlight a shared understanding across organisations of the importance of different forms of knowledge—political, professional and experiential—that are essential to the notion of ‘what works’ in practice, as well as in a theoretical or technical sense.
Beyond this shared understanding of the importance of other factors are common challenges relating to the use of evidence specifically. Some of these challenges are well documented in research on the issue in other policy fields, such as the difficulty of understanding academic literature and time constraints, which together require evidence to be presented in short, accessible forms. There are also more specific challenges characteristic of a complex and contested issue that is transitioning from being new to the policy arena to one that is in a state of flux. Contestation over definition and measurement mean that different actors seek different types of evidence according to the way in which they frame wellbeing: that is, competing theoretical conceptions of wellbeing raise different questions and require different types of answers. Moreover, there is no obvious resolution to this conflict and, indeed, disagreement over whether resolution is desirable. While it may be possible to move to a common definition in specific contexts for specific purposes, a broad consensus on these issues is not an immediate prospect. Future research might seek to explore the emerging patterns of how wellbeing is conceptualised in different contexts—either by policy area or in different political arenas—to identify any commonalities, but also to explore how different conceptions shape the nature and role of evidence in policy.
However, advocates of a greater role for wellbeing in policy view this as a long term process that competing conceptions should not deter. For many, the central purpose is to shift the focus of public policy away from a narrow conception of ‘what matters’ to people: a conception that is, in the UK at least, dominated by economic concerns and indicators. Movement in this direction has taken place without wide consensus on definitional issues, albeit with limited policy implications to date, but enough to suggest that while resolution of definitional issues might be highly desirable, it might not be essential for further movement in this direction to take place. However, there is agreement that more, high quality and accessible evidence will assist in this task: but most effectively when a broad understanding of the notion of ‘what works’ is more widely embraced.
The terms ‘individual wellbeing’ and ‘personal wellbeing’ are used interchangeably henceforth.
His report covers the use of evidence in six What Works Centre policy areas: crime reduction; health and social care; education; early intervention; ageing better; and local economic growth.
ESRC Grant Ref. ES/N003756/1. The programme team consists of five universities and five other organisations: Centre for Local Economic Strategies, Happy City, Locality, New Economics Foundation, and Social Life.
These interviews took place in August–September 2015 and were between 45 and 60 min long. The meaning of questions was clear to interviewees and no interviewees refused to answer any of the questions. The questions had been piloted with two policy actors who were part of the WWCW team.
The follow-up interviews were conducted by telephone in 2016 and 2017.
Wellbeing is relatively new to many policy areas and no systematic study has been undertaken to assess how many individuals across organisations are closely involved with wellbeing in their daily work.
Although the small sample size for organisations of different types limits the generalisability of these comparative findings.
This workshop took place on October 5–6, 2015, at Halifax Hall Hotel, University of Sheffield, Sheffield, UK.
This paper is based on research undertaken for the Community Wellbeing Evidence Programme of the What Works Centre for Wellbeing, ESRC Grant Ref. ES/N003756/1. I am very grateful to Louise Reardon and the four anonymous reviewers for SIR for their extremely helpful comments on this article. I would also like to express my thanks to colleagues in the WWCW Community Wellbeing programme team for many useful exchanges on the issues covered and to those who agreed to be interviewed for the research.
- Cabinet Office. (2015). Government guidance—What Works Network. Gov.UK, Accessed 8 September 2015 at https://www.gov.uk/guidance/what-works-network.
- Cameron, D. (2010) ‘PM speech on well-being’. Speech given by the Prime Minister, London, 25 November: http://www.number10.gov.uk/news/speeches-and-transcripts/2010/11/pm-speech-onwell-being-57569. Accessed January 2011.
- ESRC. (2014). What works centre for wellbeing 2014/2015: Common specification, Swindon: Economic and Social Research Council. www.esrc.ac.uk/_…/what-works-wellbeing-common-specification_tcm8.
- ESRC. (2017). What works centres. http://www.esrc.ac.uk/collaboration/collaboration-oportunities/what-works-centres/. Accessed 1 Sept 2017.
- Fujiwara, D., & Campbell, R. (2011). Valuation techniques for social cost-benefit analysis: Stated preference, revealed preference and subjective well-being approaches. A discussion of the current issues. London: HM Treasury, DWP.Google Scholar
- GSRU. (2007). Analysis for policy: Evidence-based policy in practice. London: Government, Social Research Unit, HM Treasury.Google Scholar
- Halperin, S., & Heath, O. (2012). Political research: Methods and skills. Oxford: Oxford University Press.Google Scholar
- HM Government. (2013). Well-being evidence submitted by the government to the Environmental Audit Committee well-being inquiry. Accessed 17 December 2015 at http://data.parliament.uk/writtenevidence/committeeevidence.svc/evidencedocument/environmental-audit-committee/wellbeing/written/1069.pdf.
- House of Commons Science and Technology Committee. (2006). Scientific advice, risk and evidence based policy making, Seventh Report of Session 2005–2006 (Vol. 1). London: The Stationery Office Limited.Google Scholar
- Labour Party. (1997). New Labour—because Britain deserves better. London: The Labour Party.Google Scholar
- Majone, G. (1989). Evidence, argument and persuasion in the policy process. New Haven, CT: Yale University Press.Google Scholar
- Marston, G., & Watts, R. (2003). Tampering with evidence: A critical appraisal of evidence-based policy-making, the drawing board. An Australian Review of Public Affairs, 3(3), 143–163.Google Scholar
- McGregor, J. A. (2015). Global initiatives in measuring human wellbeing: Convergence and difference. CWiPP Working Paper No. 2. Sheffield: Centre for Wellbeing in Public Policy, University of Sheffield.Google Scholar
- Nutley, S., Powell, A., & Davies, H. (2013). What counts as good evidence? Provocation paper for the alliance for useful evidence. Research Unit for Research Utilisation (RURRU), School of Management, University of St Andrews.Google Scholar
- O’Donnell, G., Deaton, A., Durand, D. Halpern, D., & Layard, R. (2014). Wellbeing and policy. Report Commissioned by the Legatum Institute.Google Scholar
- Self, A., Thomas, J., & Randall, C. (2012). Measuring national well-being: Life in the UK, 2012. London: Office for National Statistics.Google Scholar
- Shepherd, J. (2014). How to achieve more effective services: The evidence ecosystem—crime reduction/health and social care/education/early interventions/ageing better/local economic growth. Cardiff University/ESRC What Works Network.Google Scholar
- Weiss, C. (2001). What kind of evidence in evidence-based policy? In Third international interdisciplinary evidence-based policies and indicator systems conference, July 2001, CEM Centre, University of Durham.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.