Learning Objects and Grasp Affordances through Autonomous Exploration

  • Dirk Kraft
  • Renaud Detry
  • Nicolas Pugeault
  • Emre Başeski
  • Justus Piater
  • Norbert Krüger
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5815)


We describe a system for autonomous learning of visual object representations and their grasp affordances on a robot-vision system. It segments objects by grasping and moving 3D scene features, and creates probabilistic visual representations for object detection, recognition and pose estimation, which are then augmented by continuous characterizations of grasp affordances generated through biased, random exploration. Thus, based on a careful balance of generic prior knowledge encoded in (1) the embodiment of the system, (2) a vision system extracting structurally rich information from stereo image sequences as well as (3) a number of built-in behavioral modules on the one hand, and autonomous exploration on the other hand, the system is able to generate object and grasping knowledge through interaction with its environment.


Learn Object Object Representation Learning Cycle Unknown Object Hardware Setup 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Gibson, J.: The Ecological Approach to Visual Perception. Houghton Mifflin (1979)Google Scholar
  2. 2.
    Lowe, D.G.: Fitting parametrized 3D-models to images. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13(5), pp. 441–450 (1991)Google Scholar
  3. 3.
    Sahin, E., Cakmak, M., Dogar, M., Ugur, E., Ucoluk, G.: To afford or not to afford: A new formalization of affordances toward affordance-based robot control. Adaptive Behavior 15(4), 447–472 (2007)CrossRefGoogle Scholar
  4. 4.
    Bicchi, A., Kumar, V.: Robotic grasping and contact: A review. In: IEEE Int. Conf. on Robotics and Automation, pp. 348–353 (2000)Google Scholar
  5. 5.
    Krüger, N., Lappe, M., Wörgötter, F.: Biologically Motivated Multi-modal Processing of Visual Primitives. The Interdisciplinary Journal of Artificial Intelligence and the Simulation of Behaviour 1(5), 417–428 (2004)Google Scholar
  6. 6.
    Pugeault, N.: Early Cognitive Vision: Feedback Mechanisms for the Disambiguation of Early Visual Representation. PhD thesis, Informatics Institute, University of Göttingen (2008)Google Scholar
  7. 7.
    Pugeault, N., Wörgötter, F., Krüger, N.: Accumulated Visual Representation for Cognitive Vision. In: Proceedings of the British Machine Vision Conference, BMVC (2008)Google Scholar
  8. 8.
    Aarno, D., Sommerfeld, J., Kragic, D., Pugeault, N., Kalkan, S., Wörgötter, F., Kraft, D., Krüger, N.: Early reactive grasping with second order 3d feature relations. In: Recent Progress in Robotics: Viable Robotic Service to Human. Springer, Heidelberg (2008)Google Scholar
  9. 9.
    Detry, R., Pugeault, N., Piater, J.: A probabilistic framework for 3D visual object representation. IEEE Transactions on Pattern Analysis and Machine Intelligence (accepted, 2009)Google Scholar
  10. 10.
    Detry, R., Başeski, E., Krüger, N., Popović, M., Touati, Y., Kroemer, O., Peters, J., Piater, J.: Learning object-specific grasp affordance densities. In: International Conference on Development and Learning (to appear, 2009)Google Scholar
  11. 11.
    Kraft, D., Pugeault, N., Başeski, E., Popović, M., Kragic, D., Kalkan, S., Wörgötter, F., Krüger, N.: Birth of the Object: Detection of Objectness and Extraction of Object Shape through Object Action Complexes. Special Issue on ”Cognitive Humanoid Robots” of the International Journal of Humanoid Robotics 5, 247–265 (2009)Google Scholar
  12. 12.
    Popović, M.: An early grasping reflex in a cognitive robot vision system. Master’s thesis, The Maersk Mc-Kinney Moller Institute, University of Southern Denmark (2008)Google Scholar
  13. 13.
    Fitzpatrick, P., Metta, G.: Grounding Vision Through Experimental Manipulation. In: Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, vol. 361, pp. 2165–2185 (2003)Google Scholar
  14. 14.
    Modayil, J., Kuipers, B.: Bootstrap learning for object discovery. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), vol. 1, pp. 742–747 (2004)Google Scholar
  15. 15.
    Biegelbauer, G., Vincze, M.: Efficient 3D object detection by fitting superquadrics to range image data for robot’s object manipulation. In: IEEE International Conference on Robotics and Automation (2007)Google Scholar
  16. 16.
    Richtsfeld, M., Vincze, M.: Robotic grasping based on laser range and stereo data. In: International Conference on Robotics and Automation (2009)Google Scholar
  17. 17.
    Huebner, K., Ruthotto, S., Kragic, D.: Minimum volume bounding box decomposition for shape approximation in robot grasping. Technical report, KTH (2007)Google Scholar
  18. 18.
    Stoytchev, A.: Toward learning the binding affordances of objects: A behavior-grounded approach. In: Proceedings of AAAI Symposium on Developmental Robotics, pp. 17–22 (2005)Google Scholar
  19. 19.
    Stoytchev, A.: Learning the affordances of tools using a behavior-grounded approach. In: Rome, E., Hertzberg, J., Dorffner, G. (eds.) Towards Affordance-Based Robot Control. LNCS (LNAI), vol. 4760, pp. 140–158. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  20. 20.
    de Granville, C., Southerland, J., Fagg, A.H.: Learning grasp affordances through human demonstration. In: Proceedings of the International Conference on Development and Learning, ICDL 2006 (2006)Google Scholar
  21. 21.
    de Granville, C., Fagg, A.H.: Learning grasp affordances through human demonstration. Submitted to the Journal of Autonomous Robots (2009)Google Scholar
  22. 22.
    Baseski, E., Pugeault, N., Kalkan, S., Kraft, D., Wörgötter, F., Krüger, N.: A scene representation based on multi-modal 2D and 3D features. In: ICCV Workshop on 3D Representation for Recognition 3dRR 2007 (2007)Google Scholar
  23. 23.
    Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Francisco (1988)Google Scholar
  24. 24.
    Silverman, B.W.: Density Estimation for Statistics and Data Analysis. Chapman and Hall, CRC, Boca Raton (1986)zbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Dirk Kraft
    • 1
  • Renaud Detry
    • 2
  • Nicolas Pugeault
    • 1
  • Emre Başeski
    • 1
  • Justus Piater
    • 2
  • Norbert Krüger
    • 1
  1. 1.University of Southern DenmarkDenmark
  2. 2.University of LiègeBelgium

Personalised recommendations