Advertisement

Pragmatic-Pedagogic Value Alignment

  • Jaime F. FisacEmail author
  • Monica A. Gates
  • Jessica B. Hamrick
  • Chang Liu
  • Dylan Hadfield-Menell
  • Malayandi Palaniappan
  • Dhruv Malik
  • S. Shankar Sastry
  • Thomas L. Griffiths
  • Anca D. Dragan
Conference paper
Part of the Springer Proceedings in Advanced Robotics book series (SPAR, volume 10)

Abstract

As intelligent systems gain autonomy and capability, it becomes vital to ensure that their objectives match those of their human users; this is known as the value-alignment problem. In robotics, value alignment is key to the design of collaborative robots that can integrate into human workflows, successfully inferring and adapting to their users’ objectives as they go. We argue that a meaningful solution to value alignment must combine multi-agent decision theory with rich mathematical models of human cognition, enabling robots to tap into people’s natural collaborative capabilities. We present a solution to the cooperative inverse reinforcement learning (CIRL) dynamic game based on well-established cognitive models of decision making and theory of mind. The solution captures a key reciprocity relation: the human will not plan her actions in isolation, but rather reason pedagogically about how the robot might learn from them; the robot, in turn, can anticipate this and interpret the human’s actions pragmatically. To our knowledge, this work constitutes the first formal analysis of value alignment grounded in empirically validated cognitive models.

Keywords

Value alignment Human-robot interaction Dynamic game theory 

Notes

Acknowledgements

This work is supported by ONR under the Embedded Humans MURI (N00014-13-1-0341), by AFOSR under Implicit Communication (16RT0676), and by the Center for Human-Compatible AI.

References

  1. 1.
    Amodei, D., Steinhardt, J., Man, D., Christiano, P.: Concrete problems in AI safety (2017)Google Scholar
  2. 2.
    Hadfield-Menell, D., Dragan, A., Abbeel, P., Russell, S.: Cooperative inverse reinforcement learning. NIPS (2016)Google Scholar
  3. 3.
    Tversky, A., Kahneman, D.: Judgment under uncertainty: heuristics and biases. Science 185(4157) (1974)Google Scholar
  4. 4.
    Heider, F., Simmel, M.: An experimental study of apparent behavior. Am. J. Psychol. 57(2) (1944)Google Scholar
  5. 5.
    Meltzoff, A.N.: Understanding the intentions of others: re-enactment of intended acts by 18-month-old children. Dev. Psychol. 31(5) (1995)Google Scholar
  6. 6.
    Baker, C.L., Tenenbaum, J.B.: Modeling human plan recognition using Bayesian theory of mind. Plan Act. Intent Recognit. (2014)Google Scholar
  7. 7.
    Shafto, P., Goodman, N.D., Griffiths, T.L.: A rational account of pedagogical reasoning: teaching by, and learning from, examples. Cogn. Psychol. 71 (2014)Google Scholar
  8. 8.
    Zamir, S.: Bayesian games: games with incomplete information. Computational Complexity: Theory, Techniques, and Applications (2012)Google Scholar
  9. 9.
    Luce, R.D.: Individual Choice Behavior: A Theoretical Analysis. Wiley, New York (1959)Google Scholar
  10. 10.
    Dragan, A.D., Srinivasa, S.: Integrating human observer inferences into robot motion planning. Auton. Robot. (2014)Google Scholar
  11. 11.
    Schelling, T.C.: The Strategy of Conflict. Harvard University Press, Harvard (1960)Google Scholar
  12. 12.
    Mundhenk, M., Goldsmith, J., Lusena, C., Allender, E.: Complexity of finite-horizon Markov decision process problems. J. ACM 47(4) (2000)Google Scholar
  13. 13.
    Silver, D., Veness, J.: Monte-Carlo planning in large POMDPs. NIPS (2010)Google Scholar
  14. 14.
    Malik, D., Palaniappan, M., Fisac, J.F., Hadfield-Menell, D., Russell, S., Dragan, A. D.: An efficient, generalized Bellman update for cooperative inverse reinforcement learning. In: Dy J., Krause A. (eds.) Proceedings of the 35th International Conference on Machine Learning, vol.80, pp. 3394–3402. PMLR (2018). http://proceedings.mlr.press/v80/malik18a.html

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Jaime F. Fisac
    • 1
    Email author
  • Monica A. Gates
    • 1
  • Jessica B. Hamrick
    • 1
  • Chang Liu
    • 1
  • Dylan Hadfield-Menell
    • 1
  • Malayandi Palaniappan
    • 1
  • Dhruv Malik
    • 1
  • S. Shankar Sastry
    • 1
  • Thomas L. Griffiths
    • 1
  • Anca D. Dragan
    • 1
  1. 1.University of CaliforniaBerkeleyUSA

Personalised recommendations