New technologies and new practices, if widely and rapidly adopted, are inherently disruptive. As Joseph Schumpeter noted (1943, Ch 7), the transformations of industrial processes and business systems constitute a continuous process of ‘creative destruction’. Schumpeter’s insight about the dual face of innovation reminds us that there will always be winners and losers in system-level changes driven by technical innovation. In the early phases of industrialisation, economic transformations were driven by mechanisation and electrification. In more recent decades, these processes have been overlaid by revolutions in computerised data analysis, digital communication, machine intelligence and bio-tech engineering.
Technical inventions and patents are generally taken as the key signs of rapid innovation and economic value, but many inventions and technical adaptations do not produce social value. Public policy in a democratic society should be judged by normative criteria such as public value, social equity and environmental impacts. When innovation drives socio-economic and environmental change, the outcomes are seldom neutral. Benefits in some areas are often offset by harm or disadvantage elsewhere. Just as there have been major disruptions and uneven impacts in economic system changes, a similar pattern of disruption is evident in the impacts of industrialisation and rapid urbanisation on ecological systems and natural environments (e.g. depletion of natural resources, destruction of biodiversity habitats, pollution of air and water resources). Such environmental destruction was largely overlooked by policymaking elites for two centuries. As noted in Chapter 5, environmental policy issues have become more widely recognised and prioritised only in recent times, signalled by strategies and agreements aiming to protect ecological systems from further irreversible damage.
Advances in productivity benefit some more than others. The development of new products and services can be very profitable for investors and inventors. However, some stakeholders in economic systems are typically left behind—for example, those whose capital investments and jobs depend on systems that are rapidly becoming obsolete. Without rapid adaptation they become major losers, without a safety-net. Hence, in contemporary democracies, industry innovation and structural adjustment policies have become a higher priority for political leaders (Edler & Fagerberg, 2017), in order to mitigate the negative impacts of transitions. Modern industry policies—such as measures to accelerate a shift from fossil fuels towards renewable energy (Green & Gambhir, 2020)—accept that incentives and subsidies will be needed for two different sets of stakeholders, reflecting the two faces of innovation. Firstly, there are measures to reward and encourage entrepreneurs who are willing to invest in desirable new technologies; and secondly, other measures are designed to compensate those who cannot readily adapt because their capacities are low and transition costs are high.
In contemporary democracies, one of the great public policy challenges is to encourage innovative solutions and new agendas (Albury, 2005; Bason, 2017; Mulgan, 2014; Wanzenböck et al., 2020). Applying new technologies for social and economic improvement has become a vital commitment on the part of modern governments. But the quality of the policy system outputs should not be determined solely by novelty and innovation. It is also necessary to ensure that social value is advanced, externalities are mitigated, and the hard-won benefits of past endeavours are protected, including good governance processes. This is a difficult balancing act between the old and the new, but the quest for policy innovation should take full account of institutional knowledge and experience. In the absence of institutional knowledge and memory, novel policy options might be difficult to implement and might have unintended effects on state capacities and public trust.
In a fast-moving policy field, where risks and uncertainties are high, experiments that fail are likely to be frequent. Innovation advocates regard this as opportunities for learning (‘intelligent failure’), and opportunities for improvement through rapid cycles of trial-and-error (Hartley & Knell, 2021; Mulgan, 2014). Edmondson has argued persuasively there are many reasons for failure in social, economic and governmental decision-making. Some types of failures arise from ignorance or incompetence and are avoidable; some failures arise from well-known risks and can thus be anticipated, with procedures in place to mitigate risk. In other cases the events may be novel or unpredictable, in which case the challenge is to respond rapidly and ‘learn from failure’ (Edmondson, 2011). This approach works best at the level of small-scale experiments, where there is an acceptance that useful outcomes emerge only through iterative refinements (Cannon & Edmondson, 2005).
A number of policy scholars suggest that policy learning should be a specific goal in policy review and reform. For example, Sanderson (2009) argues that policy learning should be central in the processes for designing innovative approaches to complex and intractable policy issues. Adaptive approaches, with rapid evaluation and adjustment processes, could allow new ideas to be tested with minimal risk of negative impacts elsewhere in the system. Sanderson (2009) argues that placing a high priority on learning and continuous refinement of options can facilitate the necessary adaptations to unpredictable changes occurring in complex systems. Policy innovation to tackle wicked problems thus requires a more flexible and open mindset by policy leaders, including both public managers and their Ministers. In many countries, some key features of the political culture (e.g. bureaucratic risk-aversion, Ministerial control of policy agendas and priorities, focus on performance metrics) may hinder high-level understanding that policy solutions for wicked problems are always provisional and require continual review.
The quest for policy innovation clearly requires the development of new skills and capabilities—for example in data analytics, foresight analysis, scenario mapping and experimental design—while at the same time encouraging facilitation methods to elicit new ideas and creative thinking via multi-stakeholder processes. Ansell and Gash (2018) argue that building ‘platforms’ for ongoing collaborative discussion, design and oversight can identify pragmatic and adaptive processes for addressing complex needs under conditions of constant change. Collaborative platforms can leverage the diverse benefits of bridging and brokering organisations. They can utilise a wide range of stakeholder knowledge and can help to expand commitments to shared goals and aspirations.
In response to practical social challenges, a large number of collaborative networks have emerged in many countries, drawing upon various forms of co-funding from the social and public sectors. Examples include NESTA in the UK (https://www.nesta.org.uk/brief-history-nesta/) and TACSI in Australia (https://tacsi.org.au/about/). They work on a range of social innovation challenges—from early childhood to aged care, from poverty alleviation to skills development and from rural development to transportation efficiency in large cities. The hallmark of these organisations is their openness to many types of knowledge and their willingness to work with other networks, clearing houses and social enterprises.
The core focus of innovation design activities is on group processes, stakeholder dialogues, and a decentralised scale for problem solutions. Key methods include design labs or policy labs (Bason, 2017, McGann et al., 2018, Whicher, 2021), group workshops and digital networks that can harvest ‘collective intelligence’. Innovative design ideas emerge from dialogue among knowledge-holders who explore different perspectives and future possibilities. Recognising the impossibility of one single ‘correct’ answer, the emphasis is on identifying a few promising approaches which, in turn, require rapid testing and refinement. Stakeholder discussion tends to be anchored in specific local contexts. Many of the participants have no ambition to ‘scale up’ promising local programs for future adoption as mainstream programs at a national level. Finally, most of this ‘design’ dialogue work is conducted outside the core operations of government departments, often through consultants, think tanks and research centres working directly with citizens and stakeholder groups. This positioning may limit the impact or uptake of the policy innovation ideas by government officials and leaders unless the latter are closely involved in network steering processes (Lewis et al., 2020; van Buuren et al., 2020).
Design thinking for policy innovation may draw upon a wide range of tools, approaches and types of knowledge. The toolkit for innovative thinking is diverse, and rather different from reliance on economic cost-benefit analysis, which was a key tool for options analysis in previous decades. Design thinking approaches should be distinguished from three other contenders that purport to undertake policy innovation. The first is the long-standing quest for evidence-informed policymaking, which seeks to promote the use of best-available evidence inside the politically charged institutions of governmental decision-making (Cairney, 2016; Head, 2016). Rigorous research and analysis are expected to increase understanding about relevant trends, causal links, probable risks and likely impacts of selected interventions. Much of this research uses statistical data to explore social patterns and correlations. In a ‘rational’ policy-making process, scientific knowledge would provide expert foundations for evidence-informed solutions A second variation of evidence-rich analysis is the use of randomised controlled trials (RCTs), to test the relative efficacy of specific messages or specific adjustments to service delivery programs. The debates about the advantages and limitations of RCTs are well known, often centred on the trade-off between the analytical rigour that guarantees reliable findings, and the narrow scope of the research questions that can be the subject of tightly controlled experimentation.
A third variation is the Nudge framework (Thaler & Sunstein, 2008), as further elaborated by the Behavioural Insights Team in the UK (Halpern, 2015) and by various Behavioral Economics networks in the US (Samson, 2021). The self-limiting nature of their behavioural trials approach (micro focus on individual choices, narrow research questions for testing, and avoidance of regulatory issues) diminishes their capacity to tackle large problems. Their advocacy of try-test-learn is consistent with the pragmatic incremental outlook of Lindblom’s ‘muddling through’ (Lindblom, 1979), but their testing of non-regulatory choice options cannot rise above ‘fine-tuning’ unless they partner with other approaches that can engage with improving macro policy strategies and regulatory frameworks.
Many questions remain about the scale at which innovations need to be designed. For example, does it make sense to focus on small and manageable innovations that address part of the problem for some of the people? or do we need to focus on macro system-level changes? or develop an approach that combines all the levels? This problem is central in the ‘sustainability transitions’ literature, which encourages new thinking across all levels of the system, but also insists on connecting up every scale from local niche innovations through to strategic institutional reforms (Sengers et al., 2019; Voß et al., 2009). Challenges arise in every policy field concerning how learning can occur (Goyal & Howlett, 2020) and how small-scale initiatives can effectively contribute to tackling the complexities of wicked problems. We noted (at the end of Chapter 4) the potential value of the ‘small wins’ approach as formulated by Weick (1984) and endorsed by Termeer and Dewulf (2019).
But a ‘small wins’ approach or purpose-driven gradualism is a different space from the heated debate about experimental methodology that recently emerged concerning whether localised experimental initiatives can address entrenched inequalities. Two examples are noted here. Firstly, in about 1998 the World Bank began taking a more flexible and pluralist approach to program design for economic development and poverty alleviation. Its new interest in ‘grassroots innovation’ and experimentation led to several rounds of innovation grants, attracting hundreds of proposals. This approach also found enthusiastic support from leaders in the management innovation industry. Wood and Hamel (2002) argued that ‘big messy problems’ are not solved by ‘a few smart people’ in a policy or planning unit. These problems require that decision-makers rigorously test and refine an array of possible innovations, many of which should be sourced through grants to development stakeholders outside the traditional policy channels.
Secondly, in development economics, the award-winning research program of Banerjee and Duflo (2009, 2012) has strongly influenced the design of initiatives to address poverty and disadvantage in low-income countries. Their approach, working in cooperation with local community networks, focused on establishing trials to implement micro programs for small business. Evaluations then assessed whether these trial programs have contributed to improved skills, better access to small loans and expanded market access through new internet-based linkages. The findings of these trials then informed further refinements in the programs. However, critics have queried the validity and relevance of scientific experimentalism at a local scale, and especially queried the transferability of findings from one context to another (Deaton & Cartwright, 2018). Other critics have claimed that the impact of micro programs is necessarily very limited in regard to tackling the complex structural inequalities of power that perpetuate disadvantage and discrimination (Rodgers et al., 2020).