The Case for an Ethical Black Box

  • Alan F. T. WinfieldEmail author
  • Marina Jirotka
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10454)


This paper proposes that robots and autonomous systems should be equipped with the equivalent of a Flight Data Recorder to continuously record sensor and relevant internal status data. We call this an ethical black box. We argue that an ethical black box will be critical to the process of discovering why and how a robot caused an accident, and thus an essential part of establishing accountability and responsibility. We also argue that without the transparency afforded by an ethical black box, robots and autonomous systems are unlikely to win public trust.


Robot ethics Ethical governance Traceability Transparency Responsible robotics Trust 



This work has, in part, been supported by EPSRC grant ref EP/L024861/1. We are also grateful to the anonymous reviewers for their insightful comments.


  1. 1.
    Allen, C., Smit, I., Wallach, W.: Artificial morality: top-down, bottom-up, and hybrid approaches. Ethics Inf. Technol. 7, 149–155 (2005)CrossRefGoogle Scholar
  2. 2.
    Anderson, M., Anderson, S.L.: GenEth: a general ethical dilemma analyzer. In: Proceedings of Twenty-Eighth AAAI Conference on Artificial Intelligence, pp. 253–261 (2014)Google Scholar
  3. 3.
    Arkin, R.C., Ulam, P., Wagner, A.R.: Moral decision making in autonomous systems: enforcement, moral emotions, dignity, trust, and deception. Proc. IEEE 100(3), 571–589 (2012)CrossRefGoogle Scholar
  4. 4.
    Boden, M., Bryson, J., Caldwell, D., Dautenhahn, K., Edwards, L., Kember, S., Newman, P., Parry, V., Pegman, G., Rodden, T., Sorrell, T., Wallis, M., Whitby, B., Winfield, A.F.: Principles of robotics. Connection Sci. 29(2), 124–129 (2017)CrossRefGoogle Scholar
  5. 5.
    Bonnefon, J.-F., Shariff, A., Rahwam, I.: The social dilemma of autonomous vehicles. Sci. 352(6293), 1573–1576 (2016)Google Scholar
  6. 6.
    British Standards Institute: BS8611: 2016 Robots and robotic devices: guide to the ethical design and application of robots and robotic systems, BSI London (2016). ISBN 9780580895302Google Scholar
  7. 7.
    Campbell, N.: The evolution of flight data analysis. In: Proceedings of Australian Society of Air Safety Investigators (2007).
  8. 8.
    Choudhuri, S., Givargis, T.: FlashBox: a system for logging non-deterministic events in deployed embedded systems. In: Proceedings of 2009 ACM Symposium on Applied Computing (SAC 2009), pp. 1676–1682 (2009)Google Scholar
  9. 9.
    Elbaum, S., Munson, J.C.: Software black box: an alternative mechanism for failure analysis. In: Proceedings of 11th International Symposium on Software Reliability Engineering, ISSRE 2000, pp. 365–376 (2000)Google Scholar
  10. 10.
    Hibbard, B.: Ethical artificial intelligence, arXiv:1411.1373v9 (2014)
  11. 11.
    Caliskan-Islam, A., Bryson, J., Narayanan, A.: Semantics derived automatically from language corpora necessarily contain human biases, arXiv:1608.07187v2 (2016)
  12. 12.
    Carroll, A., Heiser, G.: An analysis of power consumption in a smartphone. In: Proceedings of 2010 USENIX Annual Technical Conference, Boston, June 2010Google Scholar
  13. 13.
    Dennis, L.A., Fisher, M., Slavkovik, M., Webster, M.: Formal verification of ethical choices in autonomous systems. Robot. Auton. Syst. 77, 1–14 (2016)CrossRefGoogle Scholar
  14. 14.
    Dillmann, R.: Benchmarks for robotics research, EURON, April 2004.
  15. 15.
    Fisher, M., List, C., Slavkovik, M., Winfield, A.F.: Engineering moral machines. In: Informatik-Spektrum. Springer, Berlin (2016)Google Scholar
  16. 16.
    Gibson, R.W.: IEEE 802 standards efforts. Comput. Netw. ISDN Syst. 19(2), 95–104 (1990)CrossRefGoogle Scholar
  17. 17.
    Grossi, D.R.: Aviation recorder overview, National Transportation Safety Board [NTSB]. J. Accid. Investig. 2(1), 31–42 (2006)Google Scholar
  18. 18.
    IEEE: Global initiative on Ethical Considerations in the Design of Artificial Intelligence and Autonomous Systems (2016).
  19. 19.
    ISO 13482:2014 Robots and robotic devices Safety requirements for personal care robots (2014).
  20. 20.
    Kavi, K.M.: Beyond the Black Box, IEEE Spectrum, Posted 30.
  21. 21.
    Menig, P., Coverdill, C.: Transportation recorders on commercial vehicles. In: Proceedings of 1999 International Symposium on Transportation Recorders (1999)Google Scholar
  22. 22.
    Moog, R.A.: MIDI: Musical Instrument Digital Interface. J. Audio Eng. Soc. 34(5), 394–404 (1986)Google Scholar
  23. 23.
    Perez, A., Garca, M.I., Nieto, M., Pedraza, J.L., Rodrguez, S., Zamorano, J.: Argos: an advanced in-vehicle data recorder on a massively sensorized vehicle for car driver behavior experimentation. IEEE Trans. Intell. Transp. Syst. 11(2), 463–473 (2010)Google Scholar
  24. 24.
    National Highway Traffic Safety Administration: Investigation Report PE 16–007 (2017).
  25. 25.
    Vlasic, B., Boudette, N.E.: Self-Driving Tesla Was Involved in Fatal Crash, U.S. Says, New York Times, 30 June 2016Google Scholar
  26. 26.
    Palmerini, E., Azzarri, F., Battaglia, A., Bertolini, A., Carnevale, A., Carpaneto, J., Cavallo, F., Di Carlo, A., Cempini, M., Controzzi, M., Koops, B.J., Lucivero, F., Mukerji, N., Nocco, L., Pirni, A., Shah, H., Salvini, P., Schellekens, M., Warwick, K.: D6.2 Guidelines on regulating robotics, Robolaw project (2014).
  27. 27.
  28. 28.
    Thom, P.R., MacCarley, C.A.: A spy under the hood: controlling risk and automotive EDR. Risk Manag. Mag. 55(2), 22–26 (2008)Google Scholar
  29. 29.
    Sharabayko, M.P., Markov, N.G.: H.264/AVC video compression on smartphones. J. Phys. Conf. Seri. 803(1) (2017). Aricle No. 012141Google Scholar
  30. 30.
    Stilgoe, J.: Tesla crash report blames human error - this is a missed opportunity, The Guardian, 21 January 2017.
  31. 31.
    Vladeck, D.C.: Machines without principals: liability rules and artificial intelligence. Wash. L. Rev. 89(117), 117–150 (2014)Google Scholar
  32. 32.
    Wilsdon, J., Willis, R.: See-through science: why public engagement needs to move upstream, DEMOS (2014)Google Scholar
  33. 33.
    Winfield, A.F.: Robotics: A Very Short Introduction, Oxford University Press, Oxford (2012)Google Scholar
  34. 34.
    Winfield, A.F.T., Blum, C., Liu, W.: Towards an ethical robot: internal models, consequences and ethical action selection. In: Mistry, M., Leonardis, A., Witkowski, M., Melhuish, C. (eds.) TAROS 2014. LNCS, vol. 8717, pp. 85–96. Springer, Cham (2014). doi: 10.1007/978-3-319-10401-0_8 Google Scholar
  35. 35.
    Winfield, A.F: Written evidence submitted to the UK Parliamentary Select Committee on Science and Technology Inquiry on Robotics and Artificial Intelligence, Discussion Paper, Science and Technology Committee (Commons) (2016)Google Scholar
  36. 36.
    Worrell, M: Analysis of Bruntingthorpe crash test data, impact. J. Inst. Traffic Accid. Investigators 21(1), 4–10 (2016)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Bristol Robotics LabUniversity of the West of EnglandBristolEngland
  2. 2.Department of Computer ScienceUniversity of OxfordOxfordEngland

Personalised recommendations