Towards Trusting Autonomous Systems

  • Michael WinikoffEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10738)


Autonomous systems are rapidly transitioning from labs into our lives. A crucial question concerns trust: in what situations will we (appropriately) trust such systems? This paper proposes three necessary prerequisites for trust. The three prerequisites are defined, motivated, and related to each other. We then consider how to realise the prerequisites. This paper aims to articulate a research agenda, and although it provides suggestions for approaches to take and directions for future work, it contains more questions than answers.



I would like to thank the anonymous reviewers for their comments, and Michael Fisher for discussions and pointers to literature.


  1. 1.
    Abbass, H.A., Petraki, E., Merrick, K., Harvey, J., Barlow, M.: Trusted autonomy and cognitive cyber symbiosis: open challenges. Cogn. Comput. 8(3), 385–408 (2016). Scholar
  2. 2.
    Arkin, R.C., Ulam, P., Wagner, A.R.: Moral decision making in autonomous systems: enforcement, moral emotions, dignity, trust, and deception. Proc. IEEE 100(3), 571–589 (2012). Scholar
  3. 3.
    Atkinson, D.J., Clark, M.H.: Autonomous agents and human interpersonal trust: can we engineer a human-machine social interface for trust? In: Trust and Autonomous Systems: Papers from the 2013 AAAI Spring Symposium, pp. 2–7 (2013)Google Scholar
  4. 4.
    Autor, D.H.: Why are there still so many jobs? The history and future of workplace automation. J. Econ. Perspect. 29(3), 3–30 (2015)CrossRefGoogle Scholar
  5. 5.
    Banavar, G.: What It Will Take for Us to Trust AI. Harvard Business Review, November 2016.
  6. 6.
    Bonnefon, J.F., Shariff, A., Rahwan, I.: The social dilemma of autonomous vehicles. Science 352(6293), 1573–1576 (2016). Scholar
  7. 7.
    Bordini, R.H., Fisher, M., Pardavila, C., Wooldridge, M.: Model checking AgentSpeak. In: Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 409–416. ACM Press (2003)Google Scholar
  8. 8.
    Bordini, R.H., Fisher, M., Wooldridge, M., Visser, W.: Property-based slicing for agent verification. J. Log. Comput. 19(6), 1385–1425 (2009). Scholar
  9. 9.
    Bordini, R.H., Hübner, J.F., Wooldridge, M.: Programming Multi-agent Systems in AgentSpeak Using Jason. Wiley (2007). ISBN 0470029005Google Scholar
  10. 10.
    Braubach, L., Pokahr, A., Lamersdorf, W.: Jadex: a BDI-agent system combining middleware and reasoning. In: Unland, R., Calisti, M., Klusch, M. (eds.) Software Agent-Based Applications, Platforms and Development Kits, pp. 143–168. Birkhäuser, Basel (2005). Scholar
  11. 11.
    Broekens, J., Harbers, M., Hindriks, K.V., van den Bosch, K., Jonker, C.M., Meyer, J.C.: Do you get it? User-evaluated explainable BDI agents. In: Dix, J., Witteveen, C. (eds.) MATES 2010. LNCS, vol. 6251, pp. 28–39. Springer, Heidelberg (2010). Scholar
  12. 12.
    Brynjolfsson, E., McAfee, A.: The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies. W.W. Norton & Company, New York (2014)Google Scholar
  13. 13.
    Caminada, M.W.A., Kutlák, R., Oren, N., Vasconcelos, W.W.: Scrutable plan enactment via argumentation and natural language generation (demonstration). In: Bazzan, A.L.C., Huhns, M.N., Lomuscio, A., Scerri, P. (eds.) International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), pp. 1625–1626. IFAAMAS (2014).
  14. 14.
    Cranefield, S., Winikoff, M., Dignum, V., Dignum, F.: No pizza for you: value-based plan selection in BDI agents. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, pp. 178–184 (2017).
  15. 15.
    Dahm, W.J.: Technology Horizons: A Vision for Air Force Science & Technology During 2010–2030. Technical report, AF/ST-TR-10-01-PR, US Air Force (2010)Google Scholar
  16. 16.
    Dastani, M.: 2APL: a practical agent programming language. Auton. Agents Multi Agent Syst. 16(3), 214–248 (2008)CrossRefGoogle Scholar
  17. 17.
    Dastani, M., Hindriks, K.V., Meyer, J.J.C. (eds.): Specification and Verification of Multi-agent Systems. Springer, Heidelberg (2010). Scholar
  18. 18.
    Dennis, L.A., Farwer, B.: Gwendolen: a BDI language for verifiable agents. In: Löwe, B. (ed.) AISB 2008 Workshop on Logic and the Simulation of Interaction and Reasoning (2008)Google Scholar
  19. 19.
    Dennis, L.A., Fisher, M., Lincoln, N.K., Lisitsa, A., Veres, S.M.: Practical verification of decision-making in agent-based autonomous systems. Autom. Softw. Eng. 23(3), 305–359 (2016). Scholar
  20. 20.
    Dennis, L.A., Fisher, M., Slavkovik, M., Webster, M.: Formal verification of ethical choices in autonomous systems. Robot. Auton. Syst. 77, 1–14 (2016). Scholar
  21. 21.
    Dennis, L.A., Fisher, M., Webster, M.P., Bordini, R.H.: Model checking agent programming languages. Autom. Softw. Eng. J. 19(1), 3–63 (2012). Scholar
  22. 22.
    Fisher, M., Dennis, L., Webster, M.: Verifying autonomous systems. Commun. ACM 56(9), 84–93 (2013)CrossRefGoogle Scholar
  23. 23.
    Fisher, M., Reed, N., Savirimuthu, J.: Misplaced trust? In: Engineering and Technology Reference. The Institution of Engineering and Technology (2015).
  24. 24.
    Gheorghiu Bobaru, M., Păsăreanu, C.S., Giannakopoulou, D.: Automated assume-guarantee reasoning by abstraction refinement. In: Gupta, A., Malik, S. (eds.) CAV 2008. LNCS, vol. 5123, pp. 135–148. Springer, Heidelberg (2008). Scholar
  25. 25.
    Gomboc, D., Solomon, S., Core, M., Lane, H.C., van Lent, M.: Design recommendations to support automated explanation and tutoring. In: Conference on Behavior Representation in Modeling and Simulation (BRIMS) (2005).
  26. 26.
    Hancock, P.A., Billings, D.R., Schaefer, K.E., Chen, J.Y.C., de Visser, E.J., Parasuraman, R.: A meta-analysis of factors affecting trust in human-robot interaction. Hum. Factors 53(5), 517–527 (2011). Scholar
  27. 27.
    Harbers, M.: Explaining Agent Behavior in Virtual Training. SIKS dissertation series no. 2011–35, SIKS (Dutch Research School for Information and Knowledge Systems) (2011)Google Scholar
  28. 28.
    Helle, P., Schamai, W., Strobel, C.: Testing of autonomous systems - challenges and current state-of-the-art. In: 26th Annual INCOSE International Symposium (2016)CrossRefGoogle Scholar
  29. 29.
    Hindriks, K.V.: Debugging is explaining. In: Rahwan, I., Wobcke, W., Sen, S., Sugawara, T. (eds.) PRIMA 2012. LNCS (LNAI), vol. 7455, pp. 31–45. Springer, Heidelberg (2012). Scholar
  30. 30.
    Jones, N.D.: An introduction to partial evaluation. ACM Comput. Surv. 28(3), 480–503 (1996). Scholar
  31. 31.
    Kaplan, J.: Artificial intelligence: think again. Commun. ACM 60(1), 36–38 (2017). Scholar
  32. 32.
    Lee, J.D., See, K.A.: Trust in automation: designing for appropriate reliance. Human Factors 46(1), 50–80 (2004)CrossRefGoogle Scholar
  33. 33.
    Li, S., Sun, W., Miller, T.: Communication in human-agent teams for tasks with joint action. In: Dignum, V., Noriega, P., Sensoy, M., Sichman, J.S.S. (eds.) COIN 2015. LNCS (LNAI), vol. 9628, pp. 224–241. Springer, Cham (2016). Scholar
  34. 34.
    Malle, B.F.: How the Mind Explains Behavior. MIT Press, Cambridge (2004). ISBN 9780262134453Google Scholar
  35. 35.
    Miller, T.: Explanation in artificial intelligence: insights from the social sciences. CoRR abs/1706.07269 (2017)Google Scholar
  36. 36.
    Pinyol, I., Sabater-Mir, J.: Computational trust and reputation models for open multi-agent systems: a review. Artif. Intell. Rev. 40(1), 1–25 (2013). Scholar
  37. 37.
    Raimondi, F., Lomuscio, A.: Automatic verification of multi-agent systems by model checking via ordered binary decision diagrams. J. Appl. Log. 5(2), 235–251 (2007)MathSciNetCrossRefGoogle Scholar
  38. 38.
    Ramchurn, S.D., Huynh, D., Jennings, N.R.: Trust in multi-agent systems. Knowl. Eng. Rev. 19(1), 1–25 (2004). Scholar
  39. 39.
    Rao, A.S., Georgeff, M.P.: BDI agents: from theory to practice. In: Lesser, V.R., Gasser, L. (eds.) Conference on Multiagent Systems, pp. 312–319. The MIT Press, San Francisco (1995)Google Scholar
  40. 40.
    van Riemsdijk, M.B., Jonker, C.M., Lesser, V.R.: Creating socially adaptive electronic partners: interaction, reasoning and ethical challenges. In: Weiss, G., Yolum, P., Bordini, R.H., Elkind, E. (eds.) Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 1201–1206. ACM (2015).
  41. 41.
    Rozier, K.Y.: Specification: the biggest bottleneck in formal methods and autonomy. In: Blazy, S., Chechik, M. (eds.) VSTTE 2016. LNCS, vol. 9971, pp. 8–26. Springer, Cham (2016). Scholar
  42. 42.
    Singh, R., Sonenberg, L., Miller, T.: Communication and sharedmental models for teams performing interdependent tasks. In: Osman, N., Sierra, C. (eds.) AAMAS 2016 Workshops, Best Papers. LNCS/LNAI, vol. 10002, pp. 163–179. Springer, Heidelberg (2016). Scholar
  43. 43.
    Stormont, D.P.: Analyzing human trust of autonomous systems in hazardous environments. In: Metzler, T. (ed.) AAAI Workshop on Human Implications of Human-Robot Interaction, pp. 27–32. The AAAI Press, Technical report WS-08-05 (2008).
  44. 44.
    Teach, R.L., Shortliffe, E.H.: An analysis of physician attitudes regarding computer-based clinical consultation systems. Comput. Biomed. Res. 14, 542–558 (1981)CrossRefGoogle Scholar
  45. 45.
    The IEEE Global Initiative for Ethical Considerations in Artificial Intelligence and Autonomous Systems: Ethically Aligned Design: A Vision For Prioritizing Wellbeing With Artificial Intelligence And Autonomous Systems, Version 1. IEEE (2016).
  46. 46.
    The White House: Artificial Intelligence, Automation, and the Economy, December 2016.
  47. 47.
    The White House: Preparing for the Future of Artificial Intelligence, October 2016.
  48. 48.
    Vesely, W.E., Goldberg, F.F., Roberts, N.H., Haasl, D.F.: Fault tree handbook. Technical report, NUREG-0492, US Nuclear Regulatory Commission, January 1981Google Scholar
  49. 49.
    Webster, M., Dixon, C., Fisher, M., Salem, M., Saunders, J., Koay, K.L., Dautenhahn, K., Saez-Pons, J.: Towards reliable autonomous robotic assistants through formal verification: a case study. IEEE Trans. Hum. Mach. Syst. 46(2), 186–196 (2016)CrossRefGoogle Scholar
  50. 50.
    Webster, M., Cameron, N., Fisher, M., Jump, M.: Generating certification evidence for autonomous unmanned aircraft using model checking and simulation. J. Aerosp. Inf. Syst. 11(5), 258–279 (2014). Scholar
  51. 51.
    Winikoff, M., Cranefield, S.: On the testability of BDI agent systems. J. Artif. Intell. Res. (JAIR) 51, 71–131 (2014). Scholar
  52. 52.
    Winikoff, M.: JACK\(^{\text{ TM }}\) intelligent agents: an industrial strength platform. In: Bordini, R.H., Dastani, M., Dix, J., Fallah-Seghrouchni, A.E. (eds.) Multi-Agent Programming: Languages, Platforms and Applications, vol. 15, pp. 175–193. Springer, Boston (2005). Scholar
  53. 53.
    Winikoff, M.: How testable are BDI agents? An analysis of branch coverage. In: Osman, N., Sierra, C. (eds.) AAMAS 2016 Workshops, Best Papers. LNCS/LNAI, vol. 10002, pp. 90–106. Springer, Cham (2016). Scholar
  54. 54.
    Winikoff, M.: Debugging agent programs with “Why?” questions. In: Das, S., Durfee, E., Larson, K., Winikoff, M. (eds.) Conference on Autonomous Agents and Multiagent Systems (AAMAS) (2017)Google Scholar
  55. 55.
    Winikoff, M., Padgham, L.: Agent oriented software engineering. In: Weiß, G. (ed.) Multiagent Systems, Chap. 15, 2 edn., pp. 695–757. MIT Press (2013)Google Scholar
  56. 56.
    Wooldridge, M., Fisher, M., Huget, M.P., Parsons, S.: Model checking multi-agent systems with MABLE. In: Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), pp. 952–959. ACM Press (2002)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Information ScienceUniversity of OtagoDunedinNew Zealand

Personalised recommendations