Intrinsically Motivated Affordance Discovery and Modeling

Abstract

In this chapter, we argue that a single intrinsic motivation function for affordance discovery can guide long-term learning in robot systems. To these ends, we provide a novel definition of “affordance” as the latent potential for the closed-loop control of environmental stimuli perceived by sensors. Specifically, the proposed intrinsic motivation function rewards the discovery of such control affordances. We will demonstrate how this function has been used by a humanoid robot to learn a number of general purpose control skills that address many different tasks. These skills, for example, include strategies for finding, grasping, and placing simple objects. We further show how this same intrinsic reward function is used to direct the robot to build stable models of when the environment affords these skills.

References

  1. Barto, A., Singh, S, Chentanez, N.: Intrinsically motivated learning of hierarchical collections of skills. In: Proceedings of the International Conference on Development and Learning (ICDL), La Jolla, CA, USA October (2004)Google Scholar
  2. Berry, D.A., Fristedt, B.: Bandit problems: Sequential allocation of experiments. In: Monographs on Statistics and Applied Probability, vol. viii + 275. Chapman & Hall, London (1985)Google Scholar
  3. Chemero, A.: An outline of a theory of affordances. Ecol. Psychol. 15(3), 181–195 (2003)CrossRefGoogle Scholar
  4. Connolly, C., Grupen, R.: Nonholonomic path planning using harmonic functions. Technical Report 94-50, University of Massachusetts, Amherst (1994)Google Scholar
  5. Detry, R., Popovic, M., Touati, Y., Baseski, E., Krüger, N, Piater, J.: Autonomous learning of object-specific grasp affordance densities. In: ICRA 2009 Workshop Approaches to Sensorimotor Learning on Humanoid Robots, Kobe, Japan (2009)Google Scholar
  6. Festinger, L.: A Theory of Cognitive Dissonance. Evanston, Row, Peterson (1957)Google Scholar
  7. Fitzpatrick, P., Metta, G., Natale, L., Rao, S., Sandini, G.: Learning about objects through action: Initial steps towards artificial cognition. In: IEEE International Conference on Robotics and Automation, Taipei, Taiwan (2003)Google Scholar
  8. Gibson, J.J.: The theory of affordances. In: Perceiving, Acting and Knowing: Toward an Ecological Psychology, pp. 67–82. Lawrence Erlbaum Associates, Hillsdale (1977)Google Scholar
  9. Harlow, H.: Learning and satiation of response in intrinsically motivated complex puzzle performances by monkeys. J. Comp. Psychol. Psychol. 43, 289–294 (1950)Google Scholar
  10. Hart, S.: The Development of Hierarchical Knowledge in Robot Systems. Ph.D. Thesis, Department of Computer Science, University of Massachusetts, Amherst (2009a)Google Scholar
  11. Hart, S.: An intrinsic reward for affordance exploration. In: Proceedings of the 8th IEEE International Conference on Development and Learning (ICDL), Shanghai, China (2009b)Google Scholar
  12. Hart, S., Grupen, R.: Natural task decomposition with intrinsic potential fields. In: Proceedings of the 2007 International Conference on Intelligent Robots and Systems (IROS), San Diego, CA (2007)Google Scholar
  13. Hart, S., Grupen, R.: Learning generalizable control programs. IEEE Trans. Auton. Mental Dev. 3(3), 216–231 (2011)CrossRefGoogle Scholar
  14. Hart, S., Sen, S., Grupen, R.: Generalization and transfer in robot control. In: 8th International Conference on Epigenetic Robotics (Epirob08), University of Sussex, Brighton, UK (2008a)Google Scholar
  15. Hart, S., Sen, S., Grupen, R.: Intrinsically motivated hierarchical manipulation. In: Proceedings of the 2008 IEEE Conference on Robots and Automation (ICRA), Pasadena, CA (2008b)Google Scholar
  16. Herrmann, J., Pawelzik, K., Geisel, T.: Learning predictive representations. Neurocomputing 32–33, 785–791 (2000)CrossRefGoogle Scholar
  17. Huang, X., Weng, J.: Novelty and reinforcement learning in the value system of developmental robots. In: Proceedings of the 2nd International Workshop on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems. Edinburgh, Scotland (2002)Google Scholar
  18. Huber, M., Grupen, R.: A hybrid discrete dynamic systems approach to robot control. Technical Report 96-43, Department of Computer Science, University of Massachusetts Amherst, Amherst (1996)Google Scholar
  19. Huber, M., MacDonald, W, Grupen, R.: A control basis for multilegged walking. In: Proceedings of the Conference on Robotics and Automation. IEEE, Minneapolis, MN, USA (1996)Google Scholar
  20. Hull, C.: Principles of Behavior: An Introduction to Behavior Theory. Appleton-Century-Croft, New York (1943)Google Scholar
  21. Hunt, H.: Intrinsic motivation and its role in psychological development. Nebraska Symp. Motiv. 13, 189–282 (1965)Google Scholar
  22. Kagan, J.: Motives and development. J. Pers. Soc. Psychol. 22, 51–66 (1972)CrossRefGoogle Scholar
  23. Koditschek, D., Rimon, E.: Robot navigation functions on manifolds with boundary. Adv. Appl. Math. 11(4), 412–442 (1990)CrossRefMathSciNetMATHGoogle Scholar
  24. Kraft, D., Pugeault, N., Baseski, E., Popović, M., Kragic, D., Kalkan, S., Wörgötter, F, Krüger, N.: Birth of the object: Detection of objectness and extraction of shape through object action complexes. In: Proceedings of the 2008 International Conference on Cognitive Systems, Karlsruhe, DE (2008)Google Scholar
  25. Krüger, N., Geib, C., Piater, J., Petrick, R., Steedman, M., Wörgötter, F., Ude, A., Asfour, T., Kraft, D., Omrcen, D., Agostini, A, Dillmann, R.: Object-action complexes: Grounded abstractions of sensori-motor processes. Robot. Auton. Syst. 59(10), 740–757 (2011)CrossRefGoogle Scholar
  26. Krüger, N., Piater, J., Wörgötter, F., Geib, C., Petrick, R., Steedman, M., Ude, A., Asfour, T., Kraft, D., Omrcen, D., Hommel, B., Agostino, A., Kragic, D., Eklundh, J., Kruger, V, Dillmann, R.: A formal definition of object action complexes and examples at different levels of the process hierarchy. http://www.paco-plus.org (2009)
  27. Modayil, J., Kupiers, B.: Autonomous development of a grounded object ontology by a learning robot. In: Proceedings of the Twenty-Second Conference on Artificial Intelligence (AAAI-07), Vancouver, British Columbia (2007)Google Scholar
  28. Montgomery, K.: The role of exploratory drive in learning. J. Comp. Psychol. Psychol. 47, 60–64 (1954)Google Scholar
  29. Nakamura, Y.: Advanced Robotics: Redundancy and Optimization. Addison-Wesley, Reading (1991)Google Scholar
  30. Oudeyer, P., Kaplan, F.: What is intrinsic motivation? a typology of computational approaches. Front. Neurorobot. 1(2) (2007)Google Scholar
  31. Oudeyer, P., Kaplan, F, Hafner, V.V.: Intrinsic motivation systems for autonomous mental development. IEEE Trans. Evol. Comput. 11, 265–286 (2007)CrossRefGoogle Scholar
  32. Piaget, J.: The Origins of Intelligence in Childhood. International Universities Press, New York (1952)CrossRefGoogle Scholar
  33. Platt, R., Fagg, A.H, Grupen, R.A.: Null space grasp control: Theory and experiments. IEEE Trans. Robot. Autom. 26, 282–295 (2010)CrossRefGoogle Scholar
  34. Quinlan, J.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo (1993)Google Scholar
  35. Rimon, E., Koditschek, D.: Exact robot navigation using artificial potential functions. IEEE Trans. Robot. Autom. 8(5), 501–518 (1992)CrossRefGoogle Scholar
  36. Şahin, E., Çakmak, M., Doǧar, M., Uǧur, E, Üçoluk, G.: To afford or not to afford: A formalization of affordances toward affordance-based robot control. Adap. Behav. 4(15), 447–472 (2007)Google Scholar
  37. Schmidhuber, J.: Adaptive curiosity and adaptive confidence. Technical Report FKI-149-91, Institut fur Informatik, Technische Universitat Munchen (1991)Google Scholar
  38. Stoytchev, A.: Toward learning the binding affordances of objects: A behavior-grounded approach. In: Proceedings of the AAAI Spring Symposium on Developmental Robotics. Stanford University, Stanford, CA (2005)Google Scholar
  39. Sutton, R., Barto, A.: Reinforcement Learning. MIT, Cambridge (1998)Google Scholar
  40. Uǧur, E., Oztop, E, Şahin, E.: Learning object affordances for planning. In: ICRA 2009 Workshop Approaches to Sensorimotor Learning on Humanoid Robots, Kobe, Japan (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.Manufacturing Systems Research LaboratoryGeneral Motors R&DWarrenUSA
  2. 2.Department of Computer ScienceUniversity of Massachusetts AmherstAmherstUSA

Personalised recommendations