Skip to main content
SpringerLink
Log in

Towards Trusting Autonomous Systems

  • Conference paper
  • First Online:
Engineering Multi-Agent Systems (EMAS 2017)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 10738))

Included in the following conference series:

Abstract

Autonomous systems are rapidly transitioning from labs into our lives. A crucial question concerns trust: in what situations will we (appropriately) trust such systems? This paper proposes three necessary prerequisites for trust. The three prerequisites are defined, motivated, and related to each other. We then consider how to realise the prerequisites. This paper aims to articulate a research agenda, and although it provides suggestions for approaches to take and directions for future work, it contains more questions than answers.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Although not all individual viewpoints receive equal prominence, which can lead to government policies being out of step with the desires of the population.

  2. 2.

    http://futureoflife.org/open-letter-autonomous-weapons/.

  3. 3.

    Partial evaluation is the process of taking a program and some of its inputs and producing a specialised program that is able to accept the remaining inputs and compute the same results as the original program, but more efficiently.

  4. 4.

    https://www.easa.europa.eu/document-library/regulations/commission-implement-ing-regulation-eu-no-9232012.

  5. 5.

    For example, that when two planes are approaching head on and there is a danger of collision, that the pilots should both turn to their right.

  6. 6.

    Other prominent BDI agent-oriented programming languages include Jason [9], Jadex [10], JACK [52], and 2APL [16].

References

  1. Abbass, H.A., Petraki, E., Merrick, K., Harvey, J., Barlow, M.: Trusted autonomy and cognitive cyber symbiosis: open challenges. Cogn. Comput. 8(3), 385–408 (2016). https://doi.org/10.1007/s12559-015-9365-5

    Article  Google Scholar 

  2. Arkin, R.C., Ulam, P., Wagner, A.R.: Moral decision making in autonomous systems: enforcement, moral emotions, dignity, trust, and deception. Proc. IEEE 100(3), 571–589 (2012). https://doi.org/10.1109/JPROC.2011.2173265

    Article  Google Scholar 

  3. Atkinson, D.J., Clark, M.H.: Autonomous agents and human interpersonal trust: can we engineer a human-machine social interface for trust? In: Trust and Autonomous Systems: Papers from the 2013 AAAI Spring Symposium, pp. 2–7 (2013)

    Google Scholar 

  4. Autor, D.H.: Why are there still so many jobs? The history and future of workplace automation. J. Econ. Perspect. 29(3), 3–30 (2015)

    Article  Google Scholar 

  5. Banavar, G.: What It Will Take for Us to Trust AI. Harvard Business Review, November 2016. https://hbr.org/2016/11/what-it-will-take-for-us-to-trust-ai

  6. Bonnefon, J.F., Shariff, A., Rahwan, I.: The social dilemma of autonomous vehicles. Science 352(6293), 1573–1576 (2016). https://doi.org/10.1126/science.aaf2654

    Article  Google Scholar 

  7. Bordini, R.H., Fisher, M., Pardavila, C., Wooldridge, M.: Model checking AgentSpeak. In: Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 409–416. ACM Press (2003)

    Google Scholar 

  8. Bordini, R.H., Fisher, M., Wooldridge, M., Visser, W.: Property-based slicing for agent verification. J. Log. Comput. 19(6), 1385–1425 (2009). https://doi.org/10.1093/logcom/exp029

    Article  MathSciNet  MATH  Google Scholar 

  9. Bordini, R.H., Hübner, J.F., Wooldridge, M.: Programming Multi-agent Systems in AgentSpeak Using Jason. Wiley (2007). ISBN 0470029005

    Google Scholar 

  10. Braubach, L., Pokahr, A., Lamersdorf, W.: Jadex: a BDI-agent system combining middleware and reasoning. In: Unland, R., Calisti, M., Klusch, M. (eds.) Software Agent-Based Applications, Platforms and Development Kits, pp. 143–168. Birkhäuser, Basel (2005). https://doi.org/10.1007/3-7643-7348-2_7

    Chapter  Google Scholar 

  11. Broekens, J., Harbers, M., Hindriks, K.V., van den Bosch, K., Jonker, C.M., Meyer, J.C.: Do you get it? User-evaluated explainable BDI agents. In: Dix, J., Witteveen, C. (eds.) MATES 2010. LNCS, vol. 6251, pp. 28–39. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-16178-0_5

    Chapter  Google Scholar 

  12. Brynjolfsson, E., McAfee, A.: The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies. W.W. Norton & Company, New York (2014)

    Google Scholar 

  13. Caminada, M.W.A., Kutlák, R., Oren, N., Vasconcelos, W.W.: Scrutable plan enactment via argumentation and natural language generation (demonstration). In: Bazzan, A.L.C., Huhns, M.N., Lomuscio, A., Scerri, P. (eds.) International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), pp. 1625–1626. IFAAMAS (2014). http://dl.acm.org/citation.cfm?id=2616095

  14. Cranefield, S., Winikoff, M., Dignum, V., Dignum, F.: No pizza for you: value-based plan selection in BDI agents. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, pp. 178–184 (2017). https://doi.org/10.24963/ijcai.2017/26

  15. Dahm, W.J.: Technology Horizons: A Vision for Air Force Science & Technology During 2010–2030. Technical report, AF/ST-TR-10-01-PR, US Air Force (2010)

    Google Scholar 

  16. Dastani, M.: 2APL: a practical agent programming language. Auton. Agents Multi Agent Syst. 16(3), 214–248 (2008)

    Article  Google Scholar 

  17. Dastani, M., Hindriks, K.V., Meyer, J.J.C. (eds.): Specification and Verification of Multi-agent Systems. Springer, Heidelberg (2010). https://doi.org/10.1007/978-1-4419-6984-2

    Book  MATH  Google Scholar 

  18. Dennis, L.A., Farwer, B.: Gwendolen: a BDI language for verifiable agents. In: Löwe, B. (ed.) AISB 2008 Workshop on Logic and the Simulation of Interaction and Reasoning (2008)

    Google Scholar 

  19. Dennis, L.A., Fisher, M., Lincoln, N.K., Lisitsa, A., Veres, S.M.: Practical verification of decision-making in agent-based autonomous systems. Autom. Softw. Eng. 23(3), 305–359 (2016). https://doi.org/10.1007/s10515-014-0168-9

    Article  Google Scholar 

  20. Dennis, L.A., Fisher, M., Slavkovik, M., Webster, M.: Formal verification of ethical choices in autonomous systems. Robot. Auton. Syst. 77, 1–14 (2016). https://doi.org/10.1016/j.robot.2015.11.012

    Article  Google Scholar 

  21. Dennis, L.A., Fisher, M., Webster, M.P., Bordini, R.H.: Model checking agent programming languages. Autom. Softw. Eng. J. 19(1), 3–63 (2012). https://doi.org/10.1007/s10515-011-0088-x

    Article  Google Scholar 

  22. Fisher, M., Dennis, L., Webster, M.: Verifying autonomous systems. Commun. ACM 56(9), 84–93 (2013)

    Article  Google Scholar 

  23. Fisher, M., Reed, N., Savirimuthu, J.: Misplaced trust? In: Engineering and Technology Reference. The Institution of Engineering and Technology (2015). https://doi.org/10.1049/etr.2014.0054

  24. Gheorghiu Bobaru, M., Păsăreanu, C.S., Giannakopoulou, D.: Automated assume-guarantee reasoning by abstraction refinement. In: Gupta, A., Malik, S. (eds.) CAV 2008. LNCS, vol. 5123, pp. 135–148. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-70545-1_14

    Chapter  Google Scholar 

  25. Gomboc, D., Solomon, S., Core, M., Lane, H.C., van Lent, M.: Design recommendations to support automated explanation and tutoring. In: Conference on Behavior Representation in Modeling and Simulation (BRIMS) (2005). http://ict.usc.edu/pubs/Design%20Recommendations%20to%20Support%20Automated%20Explanation%20and%20Tutoring.pdf

  26. Hancock, P.A., Billings, D.R., Schaefer, K.E., Chen, J.Y.C., de Visser, E.J., Parasuraman, R.: A meta-analysis of factors affecting trust in human-robot interaction. Hum. Factors 53(5), 517–527 (2011). https://doi.org/10.1177/0018720811417254

    Article  Google Scholar 

  27. Harbers, M.: Explaining Agent Behavior in Virtual Training. SIKS dissertation series no. 2011–35, SIKS (Dutch Research School for Information and Knowledge Systems) (2011)

    Google Scholar 

  28. Helle, P., Schamai, W., Strobel, C.: Testing of autonomous systems - challenges and current state-of-the-art. In: 26th Annual INCOSE International Symposium (2016)

    Article  Google Scholar 

  29. Hindriks, K.V.: Debugging is explaining. In: Rahwan, I., Wobcke, W., Sen, S., Sugawara, T. (eds.) PRIMA 2012. LNCS (LNAI), vol. 7455, pp. 31–45. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-32729-2_3

    Chapter  Google Scholar 

  30. Jones, N.D.: An introduction to partial evaluation. ACM Comput. Surv. 28(3), 480–503 (1996). https://doi.org/10.1145/243439.243447

    Article  Google Scholar 

  31. Kaplan, J.: Artificial intelligence: think again. Commun. ACM 60(1), 36–38 (2017). https://doi.org/10.1145/2950039

    Article  Google Scholar 

  32. Lee, J.D., See, K.A.: Trust in automation: designing for appropriate reliance. Human Factors 46(1), 50–80 (2004)

    Article  Google Scholar 

  33. Li, S., Sun, W., Miller, T.: Communication in human-agent teams for tasks with joint action. In: Dignum, V., Noriega, P., Sensoy, M., Sichman, J.S.S. (eds.) COIN 2015. LNCS (LNAI), vol. 9628, pp. 224–241. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-42691-4_13

    Chapter  Google Scholar 

  34. Malle, B.F.: How the Mind Explains Behavior. MIT Press, Cambridge (2004). ISBN 9780262134453

    Google Scholar 

  35. Miller, T.: Explanation in artificial intelligence: insights from the social sciences. CoRR abs/1706.07269 (2017)

    Google Scholar 

  36. Pinyol, I., Sabater-Mir, J.: Computational trust and reputation models for open multi-agent systems: a review. Artif. Intell. Rev. 40(1), 1–25 (2013). https://doi.org/10.1007/s10462-011-9277-z

    Article  Google Scholar 

  37. Raimondi, F., Lomuscio, A.: Automatic verification of multi-agent systems by model checking via ordered binary decision diagrams. J. Appl. Log. 5(2), 235–251 (2007)

    Article  MathSciNet  Google Scholar 

  38. Ramchurn, S.D., Huynh, D., Jennings, N.R.: Trust in multi-agent systems. Knowl. Eng. Rev. 19(1), 1–25 (2004). https://doi.org/10.1017/S0269888904000116

    Article  Google Scholar 

  39. Rao, A.S., Georgeff, M.P.: BDI agents: from theory to practice. In: Lesser, V.R., Gasser, L. (eds.) Conference on Multiagent Systems, pp. 312–319. The MIT Press, San Francisco (1995)

    Google Scholar 

  40. van Riemsdijk, M.B., Jonker, C.M., Lesser, V.R.: Creating socially adaptive electronic partners: interaction, reasoning and ethical challenges. In: Weiss, G., Yolum, P., Bordini, R.H., Elkind, E. (eds.) Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 1201–1206. ACM (2015). http://dl.acm.org/citation.cfm?id=2773303

  41. Rozier, K.Y.: Specification: the biggest bottleneck in formal methods and autonomy. In: Blazy, S., Chechik, M. (eds.) VSTTE 2016. LNCS, vol. 9971, pp. 8–26. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48869-1_2

    Chapter  Google Scholar 

  42. Singh, R., Sonenberg, L., Miller, T.: Communication and sharedmental models for teams performing interdependent tasks. In: Osman, N., Sierra, C. (eds.) AAMAS 2016 Workshops, Best Papers. LNCS/LNAI, vol. 10002, pp. 163–179. Springer, Heidelberg (2016). https://doi.org/10.1007/978-3-319-46882-2_10

    Chapter  Google Scholar 

  43. Stormont, D.P.: Analyzing human trust of autonomous systems in hazardous environments. In: Metzler, T. (ed.) AAAI Workshop on Human Implications of Human-Robot Interaction, pp. 27–32. The AAAI Press, Technical report WS-08-05 (2008). http://www.aaai.org/Library/Workshops/ws08-05.php

  44. Teach, R.L., Shortliffe, E.H.: An analysis of physician attitudes regarding computer-based clinical consultation systems. Comput. Biomed. Res. 14, 542–558 (1981)

    Article  Google Scholar 

  45. The IEEE Global Initiative for Ethical Considerations in Artificial Intelligence and Autonomous Systems: Ethically Aligned Design: A Vision For Prioritizing Wellbeing With Artificial Intelligence And Autonomous Systems, Version 1. IEEE (2016). http://standards.ieee.org/develop/indconn/ec/autonomous_systems.html

  46. The White House: Artificial Intelligence, Automation, and the Economy, December 2016. https://www.whitehouse.gov/sites/whitehouse.gov/files/documents/Artificial-Intelligence-Automation-Economy.PDF

  47. The White House: Preparing for the Future of Artificial Intelligence, October 2016. https://www.whitehouse.gov/sites/default/files/whitehouse_files/microsites/ostp/NSTC/preparing_for_the_future_of_ai.pdf

  48. Vesely, W.E., Goldberg, F.F., Roberts, N.H., Haasl, D.F.: Fault tree handbook. Technical report, NUREG-0492, US Nuclear Regulatory Commission, January 1981

    Google Scholar 

  49. Webster, M., Dixon, C., Fisher, M., Salem, M., Saunders, J., Koay, K.L., Dautenhahn, K., Saez-Pons, J.: Towards reliable autonomous robotic assistants through formal verification: a case study. IEEE Trans. Hum. Mach. Syst. 46(2), 186–196 (2016)

    Article  Google Scholar 

  50. Webster, M., Cameron, N., Fisher, M., Jump, M.: Generating certification evidence for autonomous unmanned aircraft using model checking and simulation. J. Aerosp. Inf. Syst. 11(5), 258–279 (2014). https://doi.org/10.2514/1.I010096

    Article  Google Scholar 

  51. Winikoff, M., Cranefield, S.: On the testability of BDI agent systems. J. Artif. Intell. Res. (JAIR) 51, 71–131 (2014). https://doi.org/10.1613/jair.4458

    Article  MathSciNet  MATH  Google Scholar 

  52. Winikoff, M.: JACK\(^{\text{ TM }}\) intelligent agents: an industrial strength platform. In: Bordini, R.H., Dastani, M., Dix, J., Fallah-Seghrouchni, A.E. (eds.) Multi-Agent Programming: Languages, Platforms and Applications, vol. 15, pp. 175–193. Springer, Boston (2005). https://doi.org/10.1007/0-387-26350-0_7

    Chapter  Google Scholar 

  53. Winikoff, M.: How testable are BDI agents? An analysis of branch coverage. In: Osman, N., Sierra, C. (eds.) AAMAS 2016 Workshops, Best Papers. LNCS/LNAI, vol. 10002, pp. 90–106. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46882-2_6

    Chapter  Google Scholar 

  54. Winikoff, M.: Debugging agent programs with “Why?” questions. In: Das, S., Durfee, E., Larson, K., Winikoff, M. (eds.) Conference on Autonomous Agents and Multiagent Systems (AAMAS) (2017)

    Google Scholar 

  55. Winikoff, M., Padgham, L.: Agent oriented software engineering. In: Weiß, G. (ed.) Multiagent Systems, Chap. 15, 2 edn., pp. 695–757. MIT Press (2013)

    Google Scholar 

  56. Wooldridge, M., Fisher, M., Huget, M.P., Parsons, S.: Model checking multi-agent systems with MABLE. In: Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), pp. 952–959. ACM Press (2002)

    Google Scholar 

Download references

Acknowledgements

I would like to thank the anonymous reviewers for their comments, and Michael Fisher for discussions and pointers to literature.

Author information

Authors and Affiliations

  1. Department of Information Science, University of Otago, Dunedin, New Zealand

    Michael Winikoff

Authors
  1. Michael Winikoff
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Winikoff .

Editor information

Editors and Affiliations

  1. LIP6, Pierre and Marie Curie University, Paris, France

    Amal El Fallah-Seghrouchni

  2. DISI, University of Bologna, Cesena, Forli, Italy

    Alessandro Ricci

  3. Department of Computer Science, New Mexico State University, Las Cruces, New Mexico, USA

    Tran Cao Son

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Winikoff, M. (2018). Towards Trusting Autonomous Systems. In: El Fallah-Seghrouchni, A., Ricci, A., Son, T. (eds) Engineering Multi-Agent Systems. EMAS 2017. Lecture Notes in Computer Science(), vol 10738. Springer, Cham. https://doi.org/10.1007/978-3-319-91899-0_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-91899-0_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-91898-3

  • Online ISBN: 978-3-319-91899-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation