What should we do when artificial intelligence (AI) goes wrong? AI has huge potential to improve the safety of societally critical systems, such as healthcare and transport, but it also has the potential to introduce new risks and amplify existing ones. For instance, biases in widely deployed diagnostic AI systems could adversely affect the care of a large number of patients (Fraser et al. 2018), and hidden weaknesses in the perception systems of autonomous vehicles may regularly expose road users to significant risk (NTSB 2019). What are the most appropriate strategies for governing the safety of AI-based systems? One answer emerges from taking contrasting looks forwards to our imagined dystopian AI future and backwards to the progressive evolution of aviation safety.

In science fiction, one of the most iconic portrayals of the control of errant AI is presented in Ridley Scott’s (and more recently Denis Villeneuve’s) Blade Runner. Based on Philip K. Dick’s dystopian novel (Dick 1968), professional ‘Blade Runners’ track down rogue humanoid AI systems which they violently ‘retire’ (destroy, or perhaps kill). Images of hazardous AI that adaptively escapes the tight confines of human control and must be covertly pursued and punitively dismantled are a common motif in popular culture (Cave and Dihal 2019). And, while fictional, the work of a Blade Runner offers an extreme illustration of one approach to the governance of AI safety. The principles underlying what might be termed ‘Blade Runner governance’ of AI safety have four key characteristics. First, it is atomised and focuses on identifying, disabling and removing a deviant individual or subsystem. Second, it is punitive and employs correctional strategies that seek accountability and retribution for prior behaviour. Third, it is compliance-oriented and focuses on deviant behaviour that breaches some pre-defined standard. Fourth, it is closed and operates through intentionally covert, hidden or secret processes. In this imagined future Blade Runners are tasked with pursuing particularly sophisticated rogue AI, but these underlying governance principles are far from fictional—they are already apparent in response to the failures and risks of current AI systems, with individual human operators blamed for the failure of complex AI (Levin 2016, 2020; Elish 2019), and a profusion of AI ethical guidelines that frame accountability as a retrospective process of determining responsibility for past failure (Jobin et al. 2019).

The problem is that these principles are contrary to much of what we know about how to improve safety in complex sociotechnical systems. A more productive and practical image to guide the governance of AI safety is not that of the Blade Runner, but is rather more prosaic, less familiar though much better understood—that of the ‘Tin Kicker’: air crash investigators who ‘kick tin’ on accident sites while picking over wreckage (Byrne 2002; Nixon and Braithwaite 2018). Professional accident and safety investigators have been central to the continuous improvement of flight safety since the dawn of aviation (Macrae 2014). The first independent air crash investigation was conducted in 1912 (Hradecky 2012), followed soon after by the establishment of the UK’s accident investigation body in 1915 (AAIB 2021). Professional safety investigation agencies have since become common in many transport sectors around the world and are emerging in other safety–critical industries like healthcare (Macrae and Vincent 2014, 2017).

AI accident investigation will be critical for building trust in AI and ensuring that AI failures are widely learnt from (Winfield and Jirotka 2017, 2018; Winfield et al. 2021). Importantly, these investigative activities will need to grapple with risks arising from the inherently sociotechnical nature of AI systems (Macrae 2019, 2021). But more fundamentally, the principles and practices that have guided the work of ‘Tin Kickers’ for over a century offer important foundational lessons for the governance of AI safety. What might be termed ‘Tin Kicker governance’ of safety illustrates a dramatic counterpoint to the principles of the AI Blade Runner. Rather than focusing on individual elements, it emphasises the systemic nature of risk, drawing on analytical methods and models that capture the complex sociotechnical processes that shape deviations in expected behaviour at all system levels, from technological to organisational to regulatory (ATSB 2007; Waterson et al. 2017). Rather than seeking accountability for past failures, it is exclusively learning-oriented and purposefully does not attribute liability or blame but instead seeks to create active accountability for future improvement (Braithwaite 2011). Rather than focusing on compliance with accepted standards, it is concerned with understanding the practical realities of complex systems, and why unexpected or deviant behaviours may be situationally rational and adaptive given particular contexts, constraints and affordances (Macrae 2014). And rather than a closed and covert process, it is fundamentally participatory, openly engaging with all relevant stakeholders to collaboratively understand reasons for failure and develop appropriate recommendations for improvement—whilst retaining authority over those findings and recommendations (Macrae and Vincent 2014).

The work of Tin Kickers, and the principles that guide this work, therefore offers a rich and productive exemplar that holds important lessons for the development of more effective strategies of AI safety governance—and is a stark contrast to the model of an AI Blade Runner that exists in the popular imagination. Indeed, Tin Kickers are already at work in AI safety, investigating the failures of some self-driving cars and automated driving systems (NTSB 2019, 2017). This work has begun to reveal some of the organisational inertia and cultural blindspots that will need to be addressed to establish more systemic, learning-oriented and participatory approaches to AI safety governance. Tesla failed to even acknowledge federal investigators’ recommendations following a fatal 2016 crash (O’Kane 2020), while Uber responded to its fatal self-driving crash by committing to build a safety management system for its test vehicles within 5 years (NTSB 2019)—a timeline seemingly as long as its manufacturing partner’s projection for large-scale deployment of self-driving cars (O’Kane 2019; Volvo 2019). These early challenges serve to emphasise the urgent importance—rather than the impossibility—of creating systems of AI safety governance that embody the principles of Tin Kickers, long before we need to resort to those of the Blade Runner.