Autonomous Robots

, 28:21 | Cite as

Generality and legibility in mobile manipulation

Learning skills for routine tasks
  • Michael Beetz
  • Freek Stulp
  • Piotr Esden-Tempski
  • Andreas Fedrizzi
  • Ulrich Klank
  • Ingo Kresse
  • Alexis Maldonado
  • Federico Ruiz


This article investigates methods for achieving more general manipulation capabilities for mobile manipulation platforms, which produce legible behavior in human living environments. To achieve generality and legibility, we combine two control mechanisms. First of all, experience- and observation-based learning of skills is applied to routine tasks, so that the repetitive and stereotypical character of everyday activity is exploited. Second, we use planning, reasoning, and search for novel tasks which have no stereotypical solution. We apply these ideas to the learning and use of action-related places, to the model-based visual recognition and localization of objects, and the learning and application of reaching strategies and motions from humans. We demonstrate the integration of these mechanisms into a single low-level control system for autonomous manipulation platforms.


Manipulation Human environments Planning Low-level control Grasping Reaching Computer vision 


  1. Albu-Schaffer, A., Haddadin, S., Ott, Ch., Stemmer, A., Wimbock, T., & Hirzinger, G. (2007). The dlr lightweight robot: design and control concepts for robots in human environments. Industrial Robot: An International Journal, 34(5), 376–385. CrossRefGoogle Scholar
  2. Asfour, T., Regenstein, K., Azad, P., Schroder, J., Bierbaum, A., Vahrenkamp, N., & Dillmann, R. (2006). Armar-iii: An integrated humanoid platform for sensory-motor control. In Proceedings of the IEEERAS/RSJ international conference on humanoid robots (Humanoids06) (pp. 169–175). Google Scholar
  3. Beetz, M., Stulp, F., Radig, B., Bandouch, J., Blodow, N., Dolha, M., Fedrizzi, A., Jain, D., Klank, U., Kresse, I., Maldonado, A., Marton, Z., Mösenlechner, L., Ruiz, F., Rusu, R. B., & Tenorth, M. (2008). The assistive kitchen—a demonstration scenario for cognitive technical systems. In IEEE 17th international symposium on robot and human interactive communication (RO-MAN), Muenchen, Germany. Invited paper. Google Scholar
  4. Belker, T. (2004). Plan projection, execution, and learning for mobile robot control. PhD thesis, Department of Applied Computer Science, University of Bonn. Google Scholar
  5. Berenson, D., Diankov, R., Nishiwaki, K., Kagami, S., & Kuffner, J. (2007). Grasp planning in complex scenes. In IEEE-RAS international conference on humanoid robots. Google Scholar
  6. Berenson, D., Srinivasa, S., Ferguson, D., Romea, A. C., & Kuffner, J. (2009). Manipulation planning with workspace goal regions. In Proceedings of the IEEE international conference on robotics and automation (ICRA). Google Scholar
  7. Billard, A., Calinon, S., Dillmann, R., & Schaal, S. (2008). Robot programming by demonstration. In Springer handbook of robotics. Berlin: Springer. Chap. 59. Google Scholar
  8. Borst, C., Fischer, M., Haidacher, S., Liu, H., & Hirzinger, G. (2003). Dlr hand ii: Experiments and experiences with an anthropomorphic hand. In 2003 IEEE international conference on robotics and automation (ICRA), Taipei, Taiwan, 14–19 September 2003. Google Scholar
  9. Calinon, S., Guenter, F., & Billard, A. (2007). On learning, representing and generalizing a task in a humanoid robot. IEEE Transactions on Systems, Man and Cybernetics, 37(2), 286–298. Special issue on robot learning by observation, demonstration and imitation. CrossRefGoogle Scholar
  10. Cheng, G., Hyon, S., Morimoto, J., Ude, A., Hale, J. G., Colvin, G., Scroggin, W., & Jacobsen, S. C. (2007). Cb: A humanoid research platform for exploring neuroscience. Journal of Advanced Robotics, 21(10), 1097–1114. CrossRefGoogle Scholar
  11. Dearden, A., & Demiris, Y. (2005). Learning forward models for robotics. In Proceedings of the nineteenth international joint conference on artificial Intelligence (IJCAI) (pp. 1440–1445). Google Scholar
  12. Fedrizzi, A., Moesenlechner, L., Stulp, F., & Beetz, M. (2009). Transformational planning for mobile manipulation based on action-related places. In Proceedings of the international conference on advanced robotics (ICAR). Google Scholar
  13. Flash, T., & Hochner, B. (2005). Motor primitives in vertebrates and invertebrates. Current Opinion in Neurobiology, 15, 660–666. CrossRefGoogle Scholar
  14. Fox, M., Gough, J., & Long, D. (2006). Using learned action models in execution monitoring. In Proceedings of UK planning and scheduling SIG. Google Scholar
  15. Gerkey, B., Vaughan, R. T., & Howard, A. (2003). The Player/Stage Project: Tools for multi-robot and distributed sensor systems. In Proceedings of the 11th international conference on advanced robotics (ICAR) (pp. 317–323). Google Scholar
  16. Haigh, K. Z. (1998). Situation-dependent learning for interleaved planning and robot execution. Ph.D. thesis, School of Computer Science, Carnegie Mellon University. Google Scholar
  17. Harris, C. M., & Wolpert, D. M. (1998). Signal-dependent noise determines motor planning. Nature, 394(20), 780–784. CrossRefGoogle Scholar
  18. Hoffmann, H., Pastor, P., Park, D.-H., & Schaal, S. (2009). Biologically-inspired dynamical systems for movement generation: Automatic real-time goal adaptation and obstacle avoidance. In IEEE international conference on robotics and automation. Google Scholar
  19. Hofhauser, A., Steger, C., & Navab, N. (2008). Harmonic deformation model for edge based template matching. In Third international conference on computer vision theory and applications (Vol. 2, pp. 75–82). Google Scholar
  20. Horswill, I. (1996). Integrated systems and naturalistic tasks. In Strategic directions in computing research AI working group. Google Scholar
  21. Ijspeert, A. J., Nakanishi, J., & Schaal, S. (2002). Movement imitation with nonlinear dynamical systems in humanoid robots. In International conference on robotics and automation (ICRA2002). Google Scholar
  22. Jenkins, O., Bodenheimer, R., & Peters, R. (2006). Manipulation manifolds: Explorations into uncovering manifolds in sensory-motor spaces. In International conference on development and learning (ICDL). Google Scholar
  23. Julier, S., & Uhlmann, J. K. (1996). A general method for approximating nonlinear transformations of probability distributions (Technical report). Dept. of Engineering Science, University of Oxford. Google Scholar
  24. Katz, D., Horrell, E., Yang, Y., Burns, B., Buckley, T., Grishkan, A., Zhylkovskyy, V., Brock, O., & Miller, E. (2006). The UMass mobile manipulator UMan: an experimental platform for autonomous mobile manipulation. In Workshop on manipulation in human environments at robotics: science and systems. Google Scholar
  25. Kemp, C., Edsinger, A., & Torres-Jara, E. (2007). Challenges for robot manipulation in human environments. IEEE Robotics and Automation Magazine, 14(1), 20–29. CrossRefGoogle Scholar
  26. Klank, U., Zia, M. Z., & Beetz, M. (2009). 3D model selection from an Internet database for robotic vision. In International conference on robotics and automation (ICRA). Google Scholar
  27. Kragic, D., Björkman, M., Christensen, H. I., & Eklundh, J. O. (2005). Vision for robotic object manipulation in domestic settings. Robotics and Autonomous Systems, 52(1), 85–100. CrossRefGoogle Scholar
  28. LaValle, S. M. (2006). Planning algorithms. Cambridge: Cambridge University Press. zbMATHGoogle Scholar
  29. Lepetit, V., & Fua, P. (2006). Keypoint recognition using randomized trees. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(9), 1465–1479. CrossRefGoogle Scholar
  30. Maciejewski, A. A., & Klein, C. A. (1989). The singular value decomposition: Computation and applications to robotics. International Journal of Robotics Research, 8(6), 63–79. CrossRefGoogle Scholar
  31. McDermott, D. (1991). A reactive plan language (Research Report YALEU/DCS/RR-864). Yale University. Google Scholar
  32. Metta, G., Fitzpatrick, P., & Natale, L. (2006). YARP: Yet Another Robot Platform. International Journal of Advanced Robotics Systems, 3(1). Special issue on Software Development and Integration in Robotics. Google Scholar
  33. Okada, K., Kojima, M., Sagawa, Y., Ichino, T., Sato, K., & Inaba, M. (2006). Vision based behavior verification system of humanoid robot for daily environment tasks. In Proceedings of the 6th IEEE-RAS international conference on humanoid robots (humanoids) (pp. 7–12). Google Scholar
  34. Osada, R., Funkhouser, T., Chazelle, B., & Dobkin, D. (2002). Shape distributions. ACM Transactions on Graphics (TOG), 21(4), 807–832. CrossRefGoogle Scholar
  35. Oztop, E., Franklin, D. W., Chaminade, T., & Cheng, G. (2005). Human-humanoid interaction: Is a humanoid robot perceived as a human? International Journal of Humanoid Robotics, 2(4), 537–559. CrossRefGoogle Scholar
  36. Park, D.-H., Hoffmann, H., & Schaal, S. (2008). Movement reproduction and obstacle avoidance with dynamic movement primitives and potential fields. In International conference on humanoid robots. Google Scholar
  37. Petrovskaya, A., & Ng, A. Y. (2007). Probabilistic mobile manipulation in dynamic environments, with application to opening doors. In International joint conference on artificial intelligence (IJCAI-07). Google Scholar
  38. Roduit, P., Martinoli, A., & Jacot, J. (2007). A quantitative method for comparing trajectories of mobile robots using point distribution models. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 2441–2448). Google Scholar
  39. Rosenbaum, D. A., Cohen, R. G., Meulenbroek, R. G. J., & Vaughan, J. (2006). Plans for grasping objects. In M. L. Latash & F. Lestienne (Eds.), Motor control and learning (pp. 9–25). New York: Springer. CrossRefGoogle Scholar
  40. Roweis, S., & Saul, L. (2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500), 2323–2326. CrossRefGoogle Scholar
  41. Rusu, R. B., Marton, Z. C., Blodow, N., Dolha, M., & Beetz, M. (2008). Towards 3D point cloud based object maps for household environments. Robotics and Autonomous Systems Journal. (Special issue on Semantic Knowledge). Google Scholar
  42. Saxena, A., Driemeyer, J., & Ng, A. Y. (2008). Robotic grasping of novel objects using vision. The International Journal of Robotics Research, 27(2), 157. CrossRefGoogle Scholar
  43. Schaal, S. (1999). Is imitation learning the route to humanoid robots? Trends in Cognitive Sciences, 3(6), 233–242. CrossRefGoogle Scholar
  44. Smits, R., De Laet, T., Claes, K., Soetens, P., De Schutter, J., & Bruyninckx, H. (2008). Orocos: A software framework for complex sensor-driven robot tasks. IEEE Robotics and Automation Magazine. Google Scholar
  45. Srinivasa, S., Ferguson, D., Weghe, M. V., Diankov, R., Berenson, D., Helfrich, C., & Strasdat, H. (2008). The robotic busboy: Steps towards developing a mobile robotic home assistant. In International conference on intelligent autonomous systems. Google Scholar
  46. Stulp, F., & Beetz, M. (2008). Refining the execution of abstract actions with learned action models. Journal of Artificial Intelligence Research (JAIR), 32, June 2008. Google Scholar
  47. Stulp, F., Koska, W., Maldonado, A., & Beetz, M. (2007). Seamless execution of action sequences. In Proceedings of the IEEE international conference on robotics and automation (ICRA) (pp. 3687–3692). Google Scholar
  48. Stulp, F., Kresse, I., Maldonado, A., Ruiz, F., Fedrizzi, A., & Beetz, M. (2009, to appear). Compact models of human reaching motions for robotic control in everyday manipulation tasks. In Proceedings of the 8th international conference on development and learning (ICDL). Google Scholar
  49. Tenorth, M., Nyga, D., & Beetz, M. (2009). Understanding and executing instructions for everyday manipulation tasks from the world wide web (Technical report). IAS Group, Technische Universität München, Fakultät für Informatik. Google Scholar
  50. Todorov, E. (2004). Optimality principles in sensorimotor control. Nature Neuroscience, 7(9), 907–915. CrossRefGoogle Scholar
  51. Wiedemann, C., Ulrich, M., & Steger, C. (2008). Recognition and tracking of 3d objects. In G. Rigoll (Ed.), Lecture notes in computer science : Vol. 5096. Pattern recognition (pp. 132–141). Berlin: Springer. CrossRefGoogle Scholar
  52. Wilhelm, A., Huissoon, J., Melek, W., Clark, C., Fuchs, M., & Hirzinger, G. (2007). Design of a wheeled mobile robotic platform with variable footprint. In IEEE/RSJ conference on intelligent robotics and systems. Google Scholar
  53. Wimmer, M., Stulp, F., Pietzsch, S., & Radig, B. (2008). Learning local objective functions for robust face model fitting. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 30(8), 1357–1370. CrossRefGoogle Scholar
  54. Wolpert, D., & Ghahramani, Z. (2000). Computational principles of movement neuroscience. Nature Neuroscience Supplement, 3, 1212–1217. CrossRefGoogle Scholar
  55. Zacharias, F., Borst, Ch., & Hirzinger, G. (2007). Capturing robot workspace structure: representing robot capabilities. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 3229–3236). Google Scholar
  56. Zacharias, F., Borst, Ch., & Hirzinger, G. (2008). Positioning mobile manipulators to perform constrained linear trajectories. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 2578–2584). Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Michael Beetz
    • 1
  • Freek Stulp
    • 1
  • Piotr Esden-Tempski
    • 1
  • Andreas Fedrizzi
    • 1
  • Ulrich Klank
    • 1
  • Ingo Kresse
    • 1
  • Alexis Maldonado
    • 1
  • Federico Ruiz
    • 1
  1. 1.Intelligent Autonomous Systems Group, Department of InformaticsTechnische Universität MünchenGarching bei MünchenGermany

Personalised recommendations