Introduction: Competing Sources of Policy Knowledge

Policy problems and challenges abound. Many of them are politically contentious. They test the strength, capacity and legitimacy of democratic governmental institutions. Some of the crises and wicked challenges arise at the global level, such as the destructive turbulence of civil wars, military conflict and trade competition. At the level of domestic policy, wicked challenges arise from structured inequalities and competing visions for social well-being, economic livelihoods and environmental protection. Understanding these problems is difficult and controversial. Calls for new thinking and policy innovation are increasingly common.

The social sciences can make large contributions to understanding and improving social and environmental problems, by working in conjunction with other stakeholders. The social sciences seem to go through cycles of over-confidence and self-doubt. In 1969, the US Social Science Research Council, reflecting on several years of ambitious Great Society programs, ruefully pointed out that the necessary knowledge was not yet available. But it was hoped that new policy programs, based on new knowledge, might emerge slowly and iteratively through pilot schemes that could be improved over time (SSRC 1969, cited in Moynihan, 1973, p. 267). Fifty years later, the same Social Science Research Council was somewhat more concerned about the rise of ‘politicization and misuse’ of social knowledge. It was also very worried about the much greater role of private corporations in producing and interpreting social data, and the gradual but steady diminution of public funding for social research. These trends posed serious risks for the reputation, integrity and perceived value of the social sciences. The SSRC called for new forms of ‘collaboration among researchers, institutions, policymakers, and the private sector, to improve the pursuit of social knowledge and its potential to contribute to the common good’ (SSRC, 2018, p. 2). However, there are no institutional mechanisms for achieving widespread collaboration, dedicated to agreed public purposes. Democratic political institutions provide a rough mechanism for balancing competing interests and objectives, but the quality of decisions will depend greatly on the skills of leaders. Political leaders who can innovate to achieve better outcomes while maintaining broad stakeholder support are relatively rare.

Ideally, the social sciences can contribute to well-informed discussion by improving the knowledge of participants in policy debates; and more directly, they can assist government leaders and officials to make sense of complex and contested issues. The capacity of democratic institutions to manage tough problems and conflicting viewpoints has become a critical test for the robustness of democracy. Building stakeholder confidence and public trust in institutions is difficult. The OECD (2017, p. 24) outlines five factors which contribute to building public trust in governmental institutions: responsiveness, reliability, integrity, openness and fairness. These five dimensions are linked to government responsibilities for providing public services, protecting citizens and using resources ethically. All aspects of trust-building require adherence to good governance principles, including the expectation that leaders should engage with a wide range of practical experience and sources of knowledge.

Very commonly, the polarised clamour of media commentators provides fertile ground for populist leaders to amplify impatient demands for certainty, in a world perceived by the general public as turbulent and threatening. Paradoxically, the era of data abundance has facilitated a greater diversity of interpretations and priorities rather than encouraged reasoned consensus. These fast-moving streams of opinion about threats and risks can leave many stakeholders feeling disgruntled, overlooked and resentful, and perhaps believing their interests and identities have been undermined in recent times. Given the pluralisation of the public and private spheres, it is all the more important to identify and negotiate ‘effective and acceptable’ policy responses. Policy knowledge and process legitimacy need to work in tandem to produce effective policy reform.

In supporting policy improvement, the social sciences do not provide a single storehouse of unified knowledge, but rather a range of methods and lenses through which complex social phenomena can be appreciated, assessed and debated. Expert knowledge is plural, ideally providing a range of complementary and informative perspectives on complex social realities. In the market-place of evidence-informed policy ideas, the social sciences can make useful contributions when researchers are attuned to significant issues and participate in discussions among practitioners and stakeholders. Social science expertise is conveyed not only through academic research centres, but also indirectly through the work of knowledge-brokering networks, think tanks, management consultancy firms and advocacy groups. There are many sources of policy innovation ideas.

In previous chapters we have noted the importance of crisis response capacities to anticipate and effectively manage disasters and sudden catastrophes (emergency management systems). We have also noted the importance of strategic foresight to inform policymakers in responding effectively to slowly evolving challenges (such as the impacts of climate change, new technologies, and the negative externalities associated with rapid industrialisation). In emerging problem areas there is low confidence in the adequacy of traditional approaches to problem-solving. Many government agencies and think tanks have become more open to encouraging new approaches. The breadth of issues on the policy agenda, including both new and old issues, has provided opportunities for re-thinking the policy innovation agenda. But the quest for new solutions needs to be progressed in ways that combine good processes (inclusive), good evidence (wide range of expertise and experience) and iterative adjustment of program delivery (adaptiveness). These factors provide the best prospects for steady improvement in outcomes under conditions of continuing change, assuming that the policy system also has adequate resources and intelligent leadership (Bryson et al., 2020). This style of policy deliberation is very different from the paradigms—rational-comprehensive planning and data-driven optimisation—that were the object of strong criticism by Rittel and Webber in their 1973 article.

There is wide interest in recent debates about how to improve policy development capacity under conditions of risk and uncertainty. The arguments for more collaborative and networked approaches to policy deliberation have already been outlined in previous chapters. The other recent approach for tackling wicked problems centres on processes to facilitate innovation and experimentation; the advocates of this approach see great potential for developing novel and effective solutions.

Innovation, Experimentation and Policy Transitions

New technologies and new practices, if widely and rapidly adopted, are inherently disruptive. As Joseph Schumpeter noted (1943, Ch 7), the transformations of industrial processes and business systems constitute a continuous process of ‘creative destruction’. Schumpeter’s insight about the dual face of innovation reminds us that there will always be winners and losers in system-level changes driven by technical innovation. In the early phases of industrialisation, economic transformations were driven by mechanisation and electrification. In more recent decades, these processes have been overlaid by revolutions in computerised data analysis, digital communication, machine intelligence and bio-tech engineering.

Technical inventions and patents are generally taken as the key signs of rapid innovation and economic value, but many inventions and technical adaptations do not produce social value. Public policy in a democratic society should be judged by normative criteria such as public value, social equity and environmental impacts. When innovation drives socio-economic and environmental change, the outcomes are seldom neutral. Benefits in some areas are often offset by harm or disadvantage elsewhere. Just as there have been major disruptions and uneven impacts in economic system changes, a similar pattern of disruption is evident in the impacts of industrialisation and rapid urbanisation on ecological systems and natural environments (e.g. depletion of natural resources, destruction of biodiversity habitats, pollution of air and water resources). Such environmental destruction was largely overlooked by policymaking elites for two centuries. As noted in Chapter 5, environmental policy issues have become more widely recognised and prioritised only in recent times, signalled by strategies and agreements aiming to protect ecological systems from further irreversible damage.

Advances in productivity benefit some more than others. The development of new products and services can be very profitable for investors and inventors. However, some stakeholders in economic systems are typically left behind—for example, those whose capital investments and jobs depend on systems that are rapidly becoming obsolete. Without rapid adaptation they become major losers, without a safety-net. Hence, in contemporary democracies, industry innovation and structural adjustment policies have become a higher priority for political leaders (Edler & Fagerberg, 2017), in order to mitigate the negative impacts of transitions. Modern industry policies—such as measures to accelerate a shift from fossil fuels towards renewable energy (Green & Gambhir, 2020)—accept that incentives and subsidies will be needed for two different sets of stakeholders, reflecting the two faces of innovation. Firstly, there are measures to reward and encourage entrepreneurs who are willing to invest in desirable new technologies; and secondly, other measures are designed to compensate those who cannot readily adapt because their capacities are low and transition costs are high.

In contemporary democracies, one of the great public policy challenges is to encourage innovative solutions and new agendas (Albury, 2005; Bason, 2017; Mulgan, 2014; Wanzenböck et al., 2020). Applying new technologies for social and economic improvement has become a vital commitment on the part of modern governments. But the quality of the policy system outputs should not be determined solely by novelty and innovation. It is also necessary to ensure that social value is advanced, externalities are mitigated, and the hard-won benefits of past endeavours are protected, including good governance processes. This is a difficult balancing act between the old and the new, but the quest for policy innovation should take full account of institutional knowledge and experience. In the absence of institutional knowledge and memory, novel policy options might be difficult to implement and might have unintended effects on state capacities and public trust.

In a fast-moving policy field, where risks and uncertainties are high, experiments that fail are likely to be frequent. Innovation advocates regard this as opportunities for learning (‘intelligent failure’), and opportunities for improvement through rapid cycles of trial-and-error (Hartley & Knell, 2021; Mulgan, 2014). Edmondson has argued persuasively there are many reasons for failure in social, economic and governmental decision-making. Some types of failures arise from ignorance or incompetence and are avoidable; some failures arise from well-known risks and can thus be anticipated, with procedures in place to mitigate risk. In other cases the events may be novel or unpredictable, in which case the challenge is to respond rapidly and ‘learn from failure’ (Edmondson, 2011). This approach works best at the level of small-scale experiments, where there is an acceptance that useful outcomes emerge only through iterative refinements (Cannon & Edmondson, 2005).

A number of policy scholars suggest that policy learning should be a specific goal in policy review and reform. For example, Sanderson (2009) argues that policy learning should be central in the processes for designing innovative approaches to complex and intractable policy issues. Adaptive approaches, with rapid evaluation and adjustment processes, could allow new ideas to be tested with minimal risk of negative impacts elsewhere in the system. Sanderson (2009) argues that placing a high priority on learning and continuous refinement of options can facilitate the necessary adaptations to unpredictable changes occurring in complex systems. Policy innovation to tackle wicked problems thus requires a more flexible and open mindset by policy leaders, including both public managers and their Ministers. In many countries, some key features of the political culture (e.g. bureaucratic risk-aversion, Ministerial control of policy agendas and priorities, focus on performance metrics) may hinder high-level understanding that policy solutions for wicked problems are always provisional and require continual review.

The quest for policy innovation clearly requires the development of new skills and capabilities—for example in data analytics, foresight analysis, scenario mapping and experimental design—while at the same time encouraging facilitation methods to elicit new ideas and creative thinking via multi-stakeholder processes. Ansell and Gash (2018) argue that building ‘platforms’ for ongoing collaborative discussion, design and oversight can identify pragmatic and adaptive processes for addressing complex needs under conditions of constant change. Collaborative platforms can leverage the diverse benefits of bridging and brokering organisations. They can utilise a wide range of stakeholder knowledge and can help to expand commitments to shared goals and aspirations.

In response to practical social challenges, a large number of collaborative networks have emerged in many countries, drawing upon various forms of co-funding from the social and public sectors. Examples include NESTA in the UK ( and TACSI in Australia ( They work on a range of social innovation challenges—from early childhood to aged care, from poverty alleviation to skills development and from rural development to transportation efficiency in large cities. The hallmark of these organisations is their openness to many types of knowledge and their willingness to work with other networks, clearing houses and social enterprises.

The core focus of innovation design activities is on group processes, stakeholder dialogues, and a decentralised scale for problem solutions. Key methods include design labs or policy labs (Bason, 2017, McGann et al., 2018, Whicher, 2021), group workshops and digital networks that can harvest ‘collective intelligence’. Innovative design ideas emerge from dialogue among knowledge-holders who explore different perspectives and future possibilities. Recognising the impossibility of one single ‘correct’ answer, the emphasis is on identifying a few promising approaches which, in turn, require rapid testing and refinement. Stakeholder discussion tends to be anchored in specific local contexts. Many of the participants have no ambition to ‘scale up’ promising local programs for future adoption as mainstream programs at a national level. Finally, most of this ‘design’ dialogue work is conducted outside the core operations of government departments, often through consultants, think tanks and research centres working directly with citizens and stakeholder groups. This positioning may limit the impact or uptake of the policy innovation ideas by government officials and leaders unless the latter are closely involved in network steering processes (Lewis et al., 2020; van Buuren et al., 2020).

Design thinking for policy innovation may draw upon a wide range of tools, approaches and types of knowledge. The toolkit for innovative thinking is diverse, and rather different from reliance on economic cost-benefit analysis, which was a key tool for options analysis in previous decades. Design thinking approaches should be distinguished from three other contenders that purport to undertake policy innovation. The first is the long-standing quest for evidence-informed policymaking, which seeks to promote the use of best-available evidence inside the politically charged institutions of governmental decision-making (Cairney, 2016; Head, 2016). Rigorous research and analysis are expected to increase understanding about relevant trends, causal links, probable risks and likely impacts of selected interventions. Much of this research uses statistical data to explore social patterns and correlations. In a ‘rational’ policy-making process, scientific knowledge would provide expert foundations for evidence-informed solutions A second variation of evidence-rich analysis is the use of randomised controlled trials (RCTs), to test the relative efficacy of specific messages or specific adjustments to service delivery programs. The debates about the advantages and limitations of RCTs are well known, often centred on the trade-off between the analytical rigour that guarantees reliable findings, and the narrow scope of the research questions that can be the subject of tightly controlled experimentation.

A third variation is the Nudge framework (Thaler & Sunstein, 2008), as further elaborated by the Behavioural Insights Team in the UK (Halpern, 2015) and by various Behavioral Economics networks in the US (Samson, 2021). The self-limiting nature of their behavioural trials approach (micro focus on individual choices, narrow research questions for testing, and avoidance of regulatory issues) diminishes their capacity to tackle large problems. Their advocacy of try-test-learn is consistent with the pragmatic incremental outlook of Lindblom’s ‘muddling through’ (Lindblom, 1979), but their testing of non-regulatory choice options cannot rise above ‘fine-tuning’ unless they partner with other approaches that can engage with improving macro policy strategies and regulatory frameworks.

Many questions remain about the scale at which innovations need to be designed. For example, does it make sense to focus on small and manageable innovations that address part of the problem for some of the people? or do we need to focus on macro system-level changes? or develop an approach that combines all the levels? This problem is central in the ‘sustainability transitions’ literature, which encourages new thinking across all levels of the system, but also insists on connecting up every scale from local niche innovations through to strategic institutional reforms (Sengers et al., 2019; Voß et al., 2009). Challenges arise in every policy field concerning how learning can occur (Goyal & Howlett, 2020) and how small-scale initiatives can effectively contribute to tackling the complexities of wicked problems. We noted (at the end of Chapter 4) the potential value of the ‘small wins’ approach as formulated by Weick (1984) and endorsed by Termeer and Dewulf (2019).

But a ‘small wins’ approach or purpose-driven gradualism is a different space from the heated debate about experimental methodology that recently emerged concerning whether localised experimental initiatives can address entrenched inequalities. Two examples are noted here. Firstly, in about 1998 the World Bank began taking a more flexible and pluralist approach to program design for economic development and poverty alleviation. Its new interest in ‘grassroots innovation’ and experimentation led to several rounds of innovation grants, attracting hundreds of proposals. This approach also found enthusiastic support from leaders in the management innovation industry. Wood and Hamel (2002) argued that ‘big messy problems’ are not solved by ‘a few smart people’ in a policy or planning unit. These problems require that decision-makers rigorously test and refine an array of possible innovations, many of which should be sourced through grants to development stakeholders outside the traditional policy channels.

Secondly, in development economics, the award-winning research program of Banerjee and Duflo (2009, 2012) has strongly influenced the design of initiatives to address poverty and disadvantage in low-income countries. Their approach, working in cooperation with local community networks, focused on establishing trials to implement micro programs for small business. Evaluations then assessed whether these trial programs have contributed to improved skills, better access to small loans and expanded market access through new internet-based linkages. The findings of these trials then informed further refinements in the programs. However, critics have queried the validity and relevance of scientific experimentalism at a local scale, and especially queried the transferability of findings from one context to another (Deaton & Cartwright, 2018). Other critics have claimed that the impact of micro programs is necessarily very limited in regard to tackling the complex structural inequalities of power that perpetuate disadvantage and discrimination (Rodgers et al., 2020).

Regulating Innovation —The Digital Economy and Biotechnologies

Rapid changes involve a mix of risks, surprises and opportunities, and in turn these generate a range of arguments about strategies and tactical responses. Fast-moving crises such as natural disasters are never welcome, but they can be anticipated to some degree, and emergency response managers can therefore initiate planning and mobilise useful resources to mitigate the impacts. On the other hand, technical innovations can emerge quickly and continuously in all areas of economic and social life. These innovations are strongly promoted by entrepreneurs who claim to be delivering benefits for targeted social groups (as well as profits for inventors and investors). The typical framing of their innovation pitch is that citizens and businesses can share the benefits and widen their choices through engaging with the new products and services. Regulatory oversight of innovations is variable, and in some cases non-existent. The dilemma for regulators is that insufficient is known at the outset about the nature and impact of novel products and services, but that once the latter have developed momentum and consumer support it becomes much more difficult for regulators to gain a mandate to restrict their operations (Moses, 2013, p. 8).

Impacts on health and safety are among the most carefully assessed and regulated. The safety and efficacy of drugs or health therapies are oversighted by regulatory bodies, with rigorous testing of the claimed medical benefits of new products. Less rigorously, the likely effects of industrial projects on environmental values are scrutinised through impact assessment regimes. In other cases, the proponents of new products and services deliberately ignore or directly challenge regulatory standards—as with the digital platform services that entered regulated markets for personalised transport. Profit-seeking companies such as Uber aggressively marketed their services directly to the public in the name of choice and competition, challenging the established taxi industry and transport regulators to block their entry into restricted and regulated markets. They then sought deregulation to allow their own operations to expand rapidly. Well-resourced innovators seek both industry disruption and policy disruption to advance their interests. Their ‘on-demand’ services were ideologically framed in terms of the ‘sharing economy’ that claims to benefit consumers and the public interest by making use of ‘idle capacity’—in this case, the vehicles owned by casual drivers. Other for-profit platforms, such as Airbnb, have entered the accommodation services business, by signing-up the owners of houses and apartments to list their facilities as short-term rental options in competition with hotels. In both these industry sectors, some jurisdictions have pushed back by requiring licenses and safety standards or by capping the volume of services available through these new channels (Spicer et al., 2019; Thelen, 2018; Tzur 2019).

The types and levels of innovation necessary for socio-economic and environmental improvements are highly variable. The standard policy approach for encouraging the technology innovation required for advancing productivity growth was to subsidise research and development (R&D) investment by private firms which are seeking market advantage and profitability. In the context of globalised competition, more recent policy frameworks have widened the analytical lens by focusing on ‘innovation-systems’ at the national level. This policy approach focused on ‘building links, clusters and networks, and on stimulating learning between elements in the systems, and enabling entrepreneurship’ (Schot & Steinmueller, 2018, p. 1554). Many questions arise as to whether these innovation-system frameworks can address the ‘transformative’ challenges entailed by sustainable development, renewable energy and climate response policies. The transitions literature suggests that multi-level innovations are required.

All forms of innovation can produce both benefits and risks. Two further examples are discussed below. Firstly, we consider big data and artificial intelligence, and secondly, biotech innovations based on gene-editing techniques in the food and health sectors.

It is widely observed that the digitisation of data has allowed a range of new applications or uses of information not available in earlier decades. Data can be analysed by computer systems for a variety of purposes. The ‘good’ uses can enhance opportunities, provide relevant information and improve client services. Other uses might entail negative risks, such as contributing to unfair or repressive treatment of individuals. Various techniques for surveillance and monitoring are a case in point—on the one hand, they can be used responsibly to protect citizens, reduce criminal behaviour and identify offenders; but on the other hand, monitoring can generate a massive data-bank of personal information that can be used by public and private organisations for profiling, targeting and commercial self-interest. Artificial intelligence (AI) systems utilise programs to analyse and interpret data within prescribed rules. They seek to optimise specified outcomes by learning from previous experiences (Nordström, 2021). The algorithms that govern digital operations are determined by organisational and commercial imperatives, and therefore the processes and outcomes should be shaped by ethical organisational leaders and governed by transparent protocols (Desouza, 2021). Ethical leadership is required for the use of AI systems in both the private and public sectors. The European Commission (2019) has published ‘ethics guidelines’ for trustworthy AI systems, outlining seven requirements: human oversight, system security and accuracy, privacy protection, transparency, non-discrimination, social and environmental value, and accountability.

The potential risks and negative impacts of digitisation and AI have attracted wide discussion, including concerns about the transformation of work (online workers, automation, robotics), the commodification of personal information for commercial marketing, and the trade-offs between enhanced community security and respect for the privacy and confidentiality of personal information. According to Winkel (2021), digitisation and AI are far from being neutral technical processes; rather, they are likely to intensify the wicked aspects of conflicting stakeholder values and perspectives arising from social complexity and uncertainty. International bodies such as the OECD have considered these concerns about the need for responsible and trustworthy innovation. The OECD (2020b) published a complex roadmap for digital transformation, similar to the European Commission guidelines, aiming to advance innovation, economic prosperity and inclusive well-being through effective regulation and good governance. Key considerations include: wide access; diverse applications; encouraging innovation and entrepreneurship; skills formation for new jobs; more effective policies to address social exclusion; enhancing trust and security; and market openness (OECD, 2020b, p. 4).

Turning to the second set of major debates about responsible innovation, we note the recent history of policies to regulate genetically modified organisms (GMOs) in the agricultural and food sector, and policies to regulate the use of gene-editing technologies for medical therapies. In the agricultural sector there is a very long history of experimentation to enhance the productivity of animals and food crops, using selective breeding techniques. The subsequent application of genetic modification (GM) techniques led to commercialisation of new crop varieties on an industrial scale, tightly protected by patents held by a few powerful US corporations (National Academies of Science, 2016). These GM products were marketed internationally with support from the US government. Many European and other countries were more circumspect and precautionary in their stance. They chose to restrict the use of GM seedstock and required product labelling for goods containing GM ingredients. Thus the framing of narratives concerning the balance of risks and opportunities was quite different in the US and elsewhere (Durant & Legge, 2006; Vogel, 2012, Ch 3). The US tended to aggressively promote domestic and international trade in GM products; the US also subsidised the inclusion of these products in foreign aid programs. European and other stakeholders deployed a broader range of impact criteria including perceived threats to rural culture and lifestyle, as well as concerns about cross-contamination and the influence of commercial monopolies. Additional types of GM technologies are being developed to address biosecurity threats, such as controlling pest species (feral animals, weeds, organisms) that cause disease, reduce productivity or displace native flora and fauna. The development of ethical guidelines for using lethal techniques to control pest species can also generate strong debate about relevant principles and various conceptions of animal welfare.

Several broader initiatives have emerged to strengthen the role of citizens and local communities in oversighting the activities of corporations that promote GM agricultural products and other potential threats to the natural environment (Gordon et al., 2021; Kuzma & Grieger, 2020). The intention is to strengthen the governance and accountability elements of the regulatory frameworks, whether those currently operating in specific countries or those proposed by international organisations. Specific industry sectors have found it necessary and useful to take community engagement seriously—for example, corporations in the extractive industries sector (mining, and gas ‘fracking’) have had to negotiate a ‘social licence to operate’ (Owen & Kemp, 2013), owing to significant disruptive impacts on farming communities. This engagement has been important for demonstrating that mining corporations can minimise risks and can distribute a range of community benefits. In many industry sectors deploying new technologies, community-focused initiatives to improve trust, transparency and accountability are being developed. These may include public dialogues about emerging technologies, and the appointment of ‘lay’ members to science advisory committees (Raman & Mohr, 2014).

Similar value-based considerations arise in the field of genome-editing technologies for improving human health outcomes. On the one hand, genomic techniques are being steadily developed to mitigate otherwise incurable conditions and to assist in new therapies to tackle cancer and degenerative diseases (Dunbar et al., 2018). On the other hand, the socially acceptable boundaries of technical innovation are having to be continually redefined in the face of unregulated and reckless experimentation, such as interventions to modify the heritable DNA of human babies (NZ Royal Society, 2019, p. 4). An expert advisory committee of the World Health Organisation has emphasised the importance of appropriate ethical standards and precautionary principles to govern human genome editing (WHO, 2021), and recommended that these standards are linked across the local, national and international levels. The common theme in all these reports and reviews is that several principles are central for oversighting biotech innovation—such as access and equity, inclusion and engagement, ethics and values, effectiveness of treatment methods and intellectual property rights. All of these add up to ‘responsible’ innovation.

Partisanship and Populism in Public Policy

The fundamental challenge for government leaders, practitioners and researchers is to identify, develop and mobilise the capacities required for addressing the policy challenges marked by complexity, diversity and uncertainty. In these endeavours, confusion and misinformation are likely to be abundant. As noted in previous Chapters, the perceived reliability of knowledge is often linked to ‘trusted sources’ of advice. These may be political-economic in nature rather than science-based. Uncertainties in knowledge are compounded by political narratives serving partisan purposes. Leaders often under-estimate problems and disparage the evidence-base. Claims about emerging threats and scenarios are necessarily speculative and become the subject of blame-games. Recent experience of turbulent crises has shown (Weible et al., 2020) that the research sector will have abundant case studies about why ‘learning’ from crises can be so fragile.

The wicked problems perspective emphasises the diverse perceptions, values and interests of stakeholders and policy actors. As outlined in Chapter 1, narratives and viewpoints about past events often inform current perceptions about likely future risks. These viewpoints largely shape how issues are defined, priorities are set, policy values are communicated and possible solutions considered. Public policy, from this viewpoint, is not so much about establishing truths but more about legitimating feasible and ‘acceptable’ next steps. Rather than a reliance on expert-driven science and data analysis, a central focus is on how to convince a majority of citizens that political leaders are providing policy leadership.

Citizens’ views are highly segmented on many issues. Government leaders therefore have to mediate between the values and interests articulated by multiple ‘publics’, namely: citizens, consumers, business lobbies, community and environmental groups. But interest-group mediation is not the full story. What if the government decides to abandon any pretence of evidence-informed diagnosis of the key issues, reject an inclusive approach to stakeholder deliberation, and instead pursue a populist agenda aligned to its partisan support base? The argument in Chapter 3 was that such a stance could be electorally popular but could undermine the long-term capacity of public governance systems to achieve sustained improvements in addressing the more difficult social, economic and environmental challenges. Partisan polarisation undermines community confidence in procedural fairness and legitimacy, which are crucial for building trust in public institutions. Polarisation inflames confirmation bias and provides excuses for ignoring inconvenient information (Nichols, 2017). Leaders are wary of accepting lessons that might reflect badly on their own competence. By contrast, long-term commitments to building good information and developing collaborative relationships can become even more valuable under crisis conditions, when these strengths need to be quickly mobilised under stress.

The populist and divisive character of public debate, reinforced by the shallow and emotive character of media communication, has influenced the quality of policy debate and leaders' behaviour across the political spectrum (Persily & Tucker, 2020). Populist leaders attempt to amplify grievances, intensify anxieties and promise quick fixes. Populism is often pitched at the people who feel resentful about their interests and identity being overlooked or displaced through social, demographic or economic change (Galston, 2018; Mudde & Kaltwasser, 2018). This leads to identity politics based on ‘us versus them’ polarisation. Populism generally rejects reliance on independent expert advice—indeed, ‘experts’ are often seen as part of the problem (e.g. their perceived elitism, or their cosmopolitan values). Populist politicians often blame ‘remote’ elites (e.g. the EU bureaucrats in Brussels, or the WTO bureaucrats in New York) for their apparent lack of policy responsiveness to the immediate needs of the people. Populist politics therefore has little space for objective scientific and professional advice (Head & Banerjee, 2020). Michael Gove expressed his frustration with ‘experts’ who raised concerns about the UK government’s case for Brexit, and President Trump frequently expressed disdain for policy advisers in government agencies and research organisations.

Public leaders wish to promote an appearance of being in control, and having answers for tough issues. However, wicked problems are inherently complex and contested, and difficult to control—whether the challenge is the Brexit transition, a refugee crisis, family violence, climate change or domestic terrorism. Hence, public leadership roles are sometimes more about navigating a turbulent sea than about reaching clear policy destinations. These uncertainties are reflected in the scholarly literature, where multiple criteria are recommended for assessing the extent of success and failure. As McConnell notes (2015, 2018), policy outcomes for complex and wicked problems are likely to be mixed, and appraisals are likely to be strongly affected by political loyalties. Progress in addressing the problems should be seen as both desirable and possible, but today’s policy decisions should be seen as provisional, and in need of review and adjustment as situations evolve.

Much of the public policy literature has maintained a pragmatic scepticism about human capacities to solve major problems. Wildavsky, for example, recognised the difficulty of tackling broad and persistent social problems: ‘problems are not so much solved as alleviated, superseded, transformed, and otherwise dropped from view’ (Wildavsky, 1979, p. 386). Problems themselves are reinvented and refurbished, just as the proposed solutions are refreshed and recycled. As Wildavsky remarked, ‘past solutions create future problems faster than present troubles can be left behind’ (Wildavsky, 1979, p.70). Policy success is seldom complete or enduring—it is more likely to be temporary, qualified and iterative. Even successful programs require adaptive management and regular evaluation.

In conclusion, improvements in the management of wicked problems are clearly possible, and some policies are more effective than others in addressing large and enduring social and environmental problems. Several factors increase the possibility that significant progress can be achieved—the quality and judgement of leaders, sufficient information, financial resources, the capacities of analysts and managers, reliable partners and cross-sector coordination. But some contextual factors are beyond the influence of governments. The contemporary era seems marked by compounding and intensifying problems which have cascading or spillover effects. There are many types of wicked problems, and they evolve in distinctive ways. Over time, some problems may diminish in intensity and become more tractable, for example, through improving the safety nets to deal with entrenched poverty. The terms in which problems are framed will shape the ‘meaningfulness’ of possible solutions. But where problems seem to become more serious over time, or where the same problem seems to be recurring with growing ferocity, more robust policy design processes are necessary, including strengthening the capacity of systems to address crises and uncertainties. In effect, public policy is a fast-moving stream in which the parameters change rapidly.