Advertisement

Towards Trusting Autonomous Systems

  • Michael WinikoffEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10738)

Abstract

Autonomous systems are rapidly transitioning from labs into our lives. A crucial question concerns trust: in what situations will we (appropriately) trust such systems? This paper proposes three necessary prerequisites for trust. The three prerequisites are defined, motivated, and related to each other. We then consider how to realise the prerequisites. This paper aims to articulate a research agenda, and although it provides suggestions for approaches to take and directions for future work, it contains more questions than answers.

Notes

Acknowledgements

I would like to thank the anonymous reviewers for their comments, and Michael Fisher for discussions and pointers to literature.

References

  1. 1.
    Abbass, H.A., Petraki, E., Merrick, K., Harvey, J., Barlow, M.: Trusted autonomy and cognitive cyber symbiosis: open challenges. Cogn. Comput. 8(3), 385–408 (2016).  https://doi.org/10.1007/s12559-015-9365-5CrossRefGoogle Scholar
  2. 2.
    Arkin, R.C., Ulam, P., Wagner, A.R.: Moral decision making in autonomous systems: enforcement, moral emotions, dignity, trust, and deception. Proc. IEEE 100(3), 571–589 (2012).  https://doi.org/10.1109/JPROC.2011.2173265CrossRefGoogle Scholar
  3. 3.
    Atkinson, D.J., Clark, M.H.: Autonomous agents and human interpersonal trust: can we engineer a human-machine social interface for trust? In: Trust and Autonomous Systems: Papers from the 2013 AAAI Spring Symposium, pp. 2–7 (2013)Google Scholar
  4. 4.
    Autor, D.H.: Why are there still so many jobs? The history and future of workplace automation. J. Econ. Perspect. 29(3), 3–30 (2015)CrossRefGoogle Scholar
  5. 5.
    Banavar, G.: What It Will Take for Us to Trust AI. Harvard Business Review, November 2016. https://hbr.org/2016/11/what-it-will-take-for-us-to-trust-ai
  6. 6.
    Bonnefon, J.F., Shariff, A., Rahwan, I.: The social dilemma of autonomous vehicles. Science 352(6293), 1573–1576 (2016).  https://doi.org/10.1126/science.aaf2654CrossRefGoogle Scholar
  7. 7.
    Bordini, R.H., Fisher, M., Pardavila, C., Wooldridge, M.: Model checking AgentSpeak. In: Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 409–416. ACM Press (2003)Google Scholar
  8. 8.
    Bordini, R.H., Fisher, M., Wooldridge, M., Visser, W.: Property-based slicing for agent verification. J. Log. Comput. 19(6), 1385–1425 (2009).  https://doi.org/10.1093/logcom/exp029MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Bordini, R.H., Hübner, J.F., Wooldridge, M.: Programming Multi-agent Systems in AgentSpeak Using Jason. Wiley (2007). ISBN 0470029005Google Scholar
  10. 10.
    Braubach, L., Pokahr, A., Lamersdorf, W.: Jadex: a BDI-agent system combining middleware and reasoning. In: Unland, R., Calisti, M., Klusch, M. (eds.) Software Agent-Based Applications, Platforms and Development Kits, pp. 143–168. Birkhäuser, Basel (2005).  https://doi.org/10.1007/3-7643-7348-2_7CrossRefGoogle Scholar
  11. 11.
    Broekens, J., Harbers, M., Hindriks, K.V., van den Bosch, K., Jonker, C.M., Meyer, J.C.: Do you get it? User-evaluated explainable BDI agents. In: Dix, J., Witteveen, C. (eds.) MATES 2010. LNCS, vol. 6251, pp. 28–39. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-16178-0_5CrossRefGoogle Scholar
  12. 12.
    Brynjolfsson, E., McAfee, A.: The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies. W.W. Norton & Company, New York (2014)Google Scholar
  13. 13.
    Caminada, M.W.A., Kutlák, R., Oren, N., Vasconcelos, W.W.: Scrutable plan enactment via argumentation and natural language generation (demonstration). In: Bazzan, A.L.C., Huhns, M.N., Lomuscio, A., Scerri, P. (eds.) International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), pp. 1625–1626. IFAAMAS (2014). http://dl.acm.org/citation.cfm?id=2616095
  14. 14.
    Cranefield, S., Winikoff, M., Dignum, V., Dignum, F.: No pizza for you: value-based plan selection in BDI agents. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, pp. 178–184 (2017).  https://doi.org/10.24963/ijcai.2017/26
  15. 15.
    Dahm, W.J.: Technology Horizons: A Vision for Air Force Science & Technology During 2010–2030. Technical report, AF/ST-TR-10-01-PR, US Air Force (2010)Google Scholar
  16. 16.
    Dastani, M.: 2APL: a practical agent programming language. Auton. Agents Multi Agent Syst. 16(3), 214–248 (2008)CrossRefGoogle Scholar
  17. 17.
    Dastani, M., Hindriks, K.V., Meyer, J.J.C. (eds.): Specification and Verification of Multi-agent Systems. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-1-4419-6984-2CrossRefzbMATHGoogle Scholar
  18. 18.
    Dennis, L.A., Farwer, B.: Gwendolen: a BDI language for verifiable agents. In: Löwe, B. (ed.) AISB 2008 Workshop on Logic and the Simulation of Interaction and Reasoning (2008)Google Scholar
  19. 19.
    Dennis, L.A., Fisher, M., Lincoln, N.K., Lisitsa, A., Veres, S.M.: Practical verification of decision-making in agent-based autonomous systems. Autom. Softw. Eng. 23(3), 305–359 (2016).  https://doi.org/10.1007/s10515-014-0168-9CrossRefGoogle Scholar
  20. 20.
    Dennis, L.A., Fisher, M., Slavkovik, M., Webster, M.: Formal verification of ethical choices in autonomous systems. Robot. Auton. Syst. 77, 1–14 (2016).  https://doi.org/10.1016/j.robot.2015.11.012CrossRefGoogle Scholar
  21. 21.
    Dennis, L.A., Fisher, M., Webster, M.P., Bordini, R.H.: Model checking agent programming languages. Autom. Softw. Eng. J. 19(1), 3–63 (2012).  https://doi.org/10.1007/s10515-011-0088-xCrossRefGoogle Scholar
  22. 22.
    Fisher, M., Dennis, L., Webster, M.: Verifying autonomous systems. Commun. ACM 56(9), 84–93 (2013)CrossRefGoogle Scholar
  23. 23.
    Fisher, M., Reed, N., Savirimuthu, J.: Misplaced trust? In: Engineering and Technology Reference. The Institution of Engineering and Technology (2015).  https://doi.org/10.1049/etr.2014.0054
  24. 24.
    Gheorghiu Bobaru, M., Păsăreanu, C.S., Giannakopoulou, D.: Automated assume-guarantee reasoning by abstraction refinement. In: Gupta, A., Malik, S. (eds.) CAV 2008. LNCS, vol. 5123, pp. 135–148. Springer, Heidelberg (2008).  https://doi.org/10.1007/978-3-540-70545-1_14CrossRefGoogle Scholar
  25. 25.
    Gomboc, D., Solomon, S., Core, M., Lane, H.C., van Lent, M.: Design recommendations to support automated explanation and tutoring. In: Conference on Behavior Representation in Modeling and Simulation (BRIMS) (2005). http://ict.usc.edu/pubs/Design%20Recommendations%20to%20Support%20Automated%20Explanation%20and%20Tutoring.pdf
  26. 26.
    Hancock, P.A., Billings, D.R., Schaefer, K.E., Chen, J.Y.C., de Visser, E.J., Parasuraman, R.: A meta-analysis of factors affecting trust in human-robot interaction. Hum. Factors 53(5), 517–527 (2011).  https://doi.org/10.1177/0018720811417254CrossRefGoogle Scholar
  27. 27.
    Harbers, M.: Explaining Agent Behavior in Virtual Training. SIKS dissertation series no. 2011–35, SIKS (Dutch Research School for Information and Knowledge Systems) (2011)Google Scholar
  28. 28.
    Helle, P., Schamai, W., Strobel, C.: Testing of autonomous systems - challenges and current state-of-the-art. In: 26th Annual INCOSE International Symposium (2016)CrossRefGoogle Scholar
  29. 29.
    Hindriks, K.V.: Debugging is explaining. In: Rahwan, I., Wobcke, W., Sen, S., Sugawara, T. (eds.) PRIMA 2012. LNCS (LNAI), vol. 7455, pp. 31–45. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-32729-2_3CrossRefGoogle Scholar
  30. 30.
    Jones, N.D.: An introduction to partial evaluation. ACM Comput. Surv. 28(3), 480–503 (1996).  https://doi.org/10.1145/243439.243447CrossRefGoogle Scholar
  31. 31.
    Kaplan, J.: Artificial intelligence: think again. Commun. ACM 60(1), 36–38 (2017).  https://doi.org/10.1145/2950039CrossRefGoogle Scholar
  32. 32.
    Lee, J.D., See, K.A.: Trust in automation: designing for appropriate reliance. Human Factors 46(1), 50–80 (2004)CrossRefGoogle Scholar
  33. 33.
    Li, S., Sun, W., Miller, T.: Communication in human-agent teams for tasks with joint action. In: Dignum, V., Noriega, P., Sensoy, M., Sichman, J.S.S. (eds.) COIN 2015. LNCS (LNAI), vol. 9628, pp. 224–241. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-42691-4_13CrossRefGoogle Scholar
  34. 34.
    Malle, B.F.: How the Mind Explains Behavior. MIT Press, Cambridge (2004). ISBN 9780262134453Google Scholar
  35. 35.
    Miller, T.: Explanation in artificial intelligence: insights from the social sciences. CoRR abs/1706.07269 (2017)Google Scholar
  36. 36.
    Pinyol, I., Sabater-Mir, J.: Computational trust and reputation models for open multi-agent systems: a review. Artif. Intell. Rev. 40(1), 1–25 (2013).  https://doi.org/10.1007/s10462-011-9277-zCrossRefGoogle Scholar
  37. 37.
    Raimondi, F., Lomuscio, A.: Automatic verification of multi-agent systems by model checking via ordered binary decision diagrams. J. Appl. Log. 5(2), 235–251 (2007)MathSciNetCrossRefGoogle Scholar
  38. 38.
    Ramchurn, S.D., Huynh, D., Jennings, N.R.: Trust in multi-agent systems. Knowl. Eng. Rev. 19(1), 1–25 (2004).  https://doi.org/10.1017/S0269888904000116CrossRefGoogle Scholar
  39. 39.
    Rao, A.S., Georgeff, M.P.: BDI agents: from theory to practice. In: Lesser, V.R., Gasser, L. (eds.) Conference on Multiagent Systems, pp. 312–319. The MIT Press, San Francisco (1995)Google Scholar
  40. 40.
    van Riemsdijk, M.B., Jonker, C.M., Lesser, V.R.: Creating socially adaptive electronic partners: interaction, reasoning and ethical challenges. In: Weiss, G., Yolum, P., Bordini, R.H., Elkind, E. (eds.) Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 1201–1206. ACM (2015). http://dl.acm.org/citation.cfm?id=2773303
  41. 41.
    Rozier, K.Y.: Specification: the biggest bottleneck in formal methods and autonomy. In: Blazy, S., Chechik, M. (eds.) VSTTE 2016. LNCS, vol. 9971, pp. 8–26. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-48869-1_2CrossRefGoogle Scholar
  42. 42.
    Singh, R., Sonenberg, L., Miller, T.: Communication and sharedmental models for teams performing interdependent tasks. In: Osman, N., Sierra, C. (eds.) AAMAS 2016 Workshops, Best Papers. LNCS/LNAI, vol. 10002, pp. 163–179. Springer, Heidelberg (2016).  https://doi.org/10.1007/978-3-319-46882-2_10CrossRefGoogle Scholar
  43. 43.
    Stormont, D.P.: Analyzing human trust of autonomous systems in hazardous environments. In: Metzler, T. (ed.) AAAI Workshop on Human Implications of Human-Robot Interaction, pp. 27–32. The AAAI Press, Technical report WS-08-05 (2008). http://www.aaai.org/Library/Workshops/ws08-05.php
  44. 44.
    Teach, R.L., Shortliffe, E.H.: An analysis of physician attitudes regarding computer-based clinical consultation systems. Comput. Biomed. Res. 14, 542–558 (1981)CrossRefGoogle Scholar
  45. 45.
    The IEEE Global Initiative for Ethical Considerations in Artificial Intelligence and Autonomous Systems: Ethically Aligned Design: A Vision For Prioritizing Wellbeing With Artificial Intelligence And Autonomous Systems, Version 1. IEEE (2016). http://standards.ieee.org/develop/indconn/ec/autonomous_systems.html
  46. 46.
    The White House: Artificial Intelligence, Automation, and the Economy, December 2016. https://www.whitehouse.gov/sites/whitehouse.gov/files/documents/Artificial-Intelligence-Automation-Economy.PDF
  47. 47.
    The White House: Preparing for the Future of Artificial Intelligence, October 2016. https://www.whitehouse.gov/sites/default/files/whitehouse_files/microsites/ostp/NSTC/preparing_for_the_future_of_ai.pdf
  48. 48.
    Vesely, W.E., Goldberg, F.F., Roberts, N.H., Haasl, D.F.: Fault tree handbook. Technical report, NUREG-0492, US Nuclear Regulatory Commission, January 1981Google Scholar
  49. 49.
    Webster, M., Dixon, C., Fisher, M., Salem, M., Saunders, J., Koay, K.L., Dautenhahn, K., Saez-Pons, J.: Towards reliable autonomous robotic assistants through formal verification: a case study. IEEE Trans. Hum. Mach. Syst. 46(2), 186–196 (2016)CrossRefGoogle Scholar
  50. 50.
    Webster, M., Cameron, N., Fisher, M., Jump, M.: Generating certification evidence for autonomous unmanned aircraft using model checking and simulation. J. Aerosp. Inf. Syst. 11(5), 258–279 (2014).  https://doi.org/10.2514/1.I010096CrossRefGoogle Scholar
  51. 51.
    Winikoff, M., Cranefield, S.: On the testability of BDI agent systems. J. Artif. Intell. Res. (JAIR) 51, 71–131 (2014).  https://doi.org/10.1613/jair.4458MathSciNetCrossRefzbMATHGoogle Scholar
  52. 52.
    Winikoff, M.: JACK\(^{\text{ TM }}\) intelligent agents: an industrial strength platform. In: Bordini, R.H., Dastani, M., Dix, J., Fallah-Seghrouchni, A.E. (eds.) Multi-Agent Programming: Languages, Platforms and Applications, vol. 15, pp. 175–193. Springer, Boston (2005).  https://doi.org/10.1007/0-387-26350-0_7CrossRefGoogle Scholar
  53. 53.
    Winikoff, M.: How testable are BDI agents? An analysis of branch coverage. In: Osman, N., Sierra, C. (eds.) AAMAS 2016 Workshops, Best Papers. LNCS/LNAI, vol. 10002, pp. 90–106. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46882-2_6CrossRefGoogle Scholar
  54. 54.
    Winikoff, M.: Debugging agent programs with “Why?” questions. In: Das, S., Durfee, E., Larson, K., Winikoff, M. (eds.) Conference on Autonomous Agents and Multiagent Systems (AAMAS) (2017)Google Scholar
  55. 55.
    Winikoff, M., Padgham, L.: Agent oriented software engineering. In: Weiß, G. (ed.) Multiagent Systems, Chap. 15, 2 edn., pp. 695–757. MIT Press (2013)Google Scholar
  56. 56.
    Wooldridge, M., Fisher, M., Huget, M.P., Parsons, S.: Model checking multi-agent systems with MABLE. In: Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), pp. 952–959. ACM Press (2002)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Information ScienceUniversity of OtagoDunedinNew Zealand

Personalised recommendations