Observation and Execution

  • Christoph Borst
  • Franziska Zacharias
  • Florian Schmidt
  • Daniel Leidner
  • Maximo A. Roa
  • Katharina Hertkorn
  • Gerhard Grunwald
  • Pietro Falco
  • Ciro Natale
  • Emilio Maggio
Part of the Springer Tracts in Advanced Robotics book series (STAR, volume 80)


Assistive robotic systems in household or industrial production environments get more and more capable of performing also complex tasks which previously only humans were able to do. As robots are often equipped with two arms and hands, similar manipulations can be executed. The robust programming of such devices with a very large number of degrees of freedom (DOFs) compared with single industrial robot arms however is laborious if done joint-wise. Two major directions to overcome this problem have been previously proposed. The programming by demonstration (PbD) approach, where human arm and recently also hand motions are tracked, segmented and re-executed in an adaptive way on the robotic system and the high-level planning approach which tries to generate a task sequence on a logical level and attributes geometric information as necessary to generate artificial trajectories to solve the task. Here we propose to combine the best of both worlds. For the very complex motion generation for a robotic hand, a rather direct approach to assign manipulation actions from human demonstration to a human hand is taken. For the combination of different basic manipulation actions the task constraints are segmented from the demonstration action and used to generate a task oriented plan. This plan is validated against the robot kinematic and geometric constraints and then a geometric motion planner can generate the necessary robot motions to fulfill the task execution on the system.


Root Mean Square Error Joint Angle Humanoid Robot Inverse Kinematic Mobile Manipulator 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abdel-Malek, K., Yu, W.: Placement of robot manipulators to maximize dexterity. Journal of Robotics and Automation 19, 6–15 (2004)Google Scholar
  2. 2.
    Berenson, D., Srinivasa, S., Kuffner, J.: Addressing pose uncertainty in manipulation planning using task space regions. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Saint Louis, MO (2009)Google Scholar
  3. 3.
    Bronstein, I.N., Semendjajew, K.A., Musiol, G., Mühlig, H.: Taschenbuch der Mathematik, 6th edn. Harri Deutsch (2005)Google Scholar
  4. 4.
    Butterfass, J., Grebenstein, M., Liu, H., Hirzinger, G.: DLR hand II: Next generation of a dextrous robot hand. In: IEEE International Conference on Robotics and Automation, Seoul (2001)Google Scholar
  5. 5.
    Cavallo, A., De Maria, G., Natale, C., Pirozzi, S.: Optoelectronic joint angular sensor for robotic fingers. Sensors and Actuators A: Physical 152, 203–210 (2009)CrossRefGoogle Scholar
  6. 6.
    Cerveri, P., Lopomo, N., Pedotti, A., Ferrigno, G.: Derivation of centers and axes of rotation for wrist and finger in a hand kinematic model: Methods and reliability results. Annals of Biomechanics Engineering 33, 402–412 (2005)CrossRefGoogle Scholar
  7. 7.
    Cerveri, P., Pedotti, A., Ferrigno, G.: Robust recovery of human motion from video using kalman filters and virtual humans. Human Movement Science 22, 377–404 (2003)CrossRefGoogle Scholar
  8. 8.
    Chang, L.Y., Pollard, N.: Method for determining kinematic parameters of the in vivo thumb carpometacarpal joint. IEEE Transactions on Biomedical Engineering 55, 1897–1906 (2008)CrossRefGoogle Scholar
  9. 9.
    Corato, F., Falco, P., Lösch, M., Maggio, E., Jäkel, R., Villani, L.: Original approaches to interpretation, learning and modeling, from the observation of human manipulation. In: Robotics: Science and Systems, Workshop on Understanding the Human Hand for Advancing Robotic Manipulation, Seattle, WA (2009)Google Scholar
  10. 10.
    Craig, J.: Introduction to Robotics: Mechanics and Control. Addison-Wesley, Reading (1989)zbMATHGoogle Scholar
  11. 11.
    De Maria, G., Falco, P., Natale, C., Pirozzi, S.: Data fusion based on optical technology for observation of human manipulation. International Journal on Optomechatronics 6(1) (2012)Google Scholar
  12. 12.
    Denavit, J., Hartenberg, R.: A kinematic notation for lower-pair mechanisms based on matrices. Journal of Applied Mechanics 7, 215–221 (1955)MathSciNetGoogle Scholar
  13. 13.
    Diankov, R.: Automated construction of robotic manipulation programs, Ph.D. thesis. Carnegie Mellon University (2010)Google Scholar
  14. 14.
    Diankov, R., Ratliff, N., Ferguson, D., Srinivasa, S., Kuffner, J.: Bispace planning: Concurrent multi-space exploration. In: Robotics: Science and Systems, Zurich (2008)Google Scholar
  15. 15.
    Diankov, R., Srinivasa, S., Ferguson, D., Kuffner, J.: Manipulation planning with caging grasps. In: 8th IEEE-RAS International Conference on Humanoid Robots, Daejeon (2008)Google Scholar
  16. 16.
    Dorfmuller-Ulhaas, K., Schmalstieg, D.: Finger tracking for interaction in augmented environments. In: IEEE/ACM International Symposium on Augmented Reality, New York (2001)Google Scholar
  17. 17.
    Falco, P., Natale, C.: On the stability of closed-loop inverse kinematics algorithms for redundant robots. IEEE Transactions on Robotics 27, 780–784 (2011)CrossRefGoogle Scholar
  18. 18.
    Fuchs, M., Borst, C., Giordano, P.R., Baumann, A., Krämer, E., Langwald, J., Gruber, R., Seitz, N., Plank, G., Kunze, K., Burger, R., Schmidt, F., Wimboeck, T., Hirzinger, G.: Rollin’ Justin — Design considerations and realization of a mobile platform for a humanoid upper body. In: IEEE International Conference on Robotics and Automation, Kobe (2009)Google Scholar
  19. 19.
    Gienger, M., Toussaint, M., Goerick, C.: Task maps in humanoid robot manipulation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Nice (2008)Google Scholar
  20. 20.
    Gienger, M., Toussaint, M., Jetchev, N., Bendig, A., Goerick, C.: Optimization of fluent approach and grasp motions. In: 8th IEEE-RAS International Conference on Humanoid Robots, Daejeon (2008)Google Scholar
  21. 21.
    Goldfeder, C., Ciocarlie, M., Dang, H., Allen, P.K.: The Columbia grasp database. In: IEEE International Conference on Robotics and Automation, Kobe (2009)Google Scholar
  22. 22.
    Guan, Y., Yokoi, K.: Reachable space generation of a humanoid robot using the monte carlo method. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Beijing (2006)Google Scholar
  23. 23.
    Han, L., Trinkle, J., Li, Z.: Grasp analysis as linear matrix inequality problems. IEEE Transactions on Robotics and Automation 16, 663–674 (2000)CrossRefGoogle Scholar
  24. 24.
    Jung, S., Wohn, K.: Tracking and motion estimation of the articulated object: A hierarchical Kalman filter approach. Real-Time Imaging 3, 415–432 (1997)CrossRefGoogle Scholar
  25. 25.
    Klein, C.A., Blaho, B.E.: Dexterity measures for the design and control of kinematically redundant manipulators. International Journal of Robotics Research 6(2), 72–83 (1987)CrossRefGoogle Scholar
  26. 26.
    Konietschke, R., Frumento, S., Ortmaier, T., Hagn, U., Hirzinger, G.: Kinematic design optimization of an actuated carrier for the DLR multi-arm surgical system. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Beijing (2006)Google Scholar
  27. 27.
    Larsen, E., Gottschalk, S., Lin, M.C., Manocha, D.: Fast proximity queries with swept sphere volumes. In: IEEE International Conference on Robotics and Automation, San Francisco, CA (2000)Google Scholar
  28. 28.
    Liu, Y.: Qualitative test and force optimization of 3-D frictional form-closure grasps using linear programming. IEEE Transactions on Robotics and Automation 15, 163–173 (1999)CrossRefGoogle Scholar
  29. 29.
    Miyata, N., Kouhci, M., Kurihara, T., Mochimaru, M.: Modeling of human hand link structure from optical motion capture data. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Sendai (2004)Google Scholar
  30. 30.
    Murray, R., Li, Z., Sastry, S.: A Mathematical Introduction to Robotic Manipulation. CRC Press, Boca Raton (1994)zbMATHGoogle Scholar
  31. 31.
    Natale, C.: Kinematic control of robots with noisy guidance systems. In: 18th IFAC World Congress, Milan (2011)Google Scholar
  32. 32.
    Ott, C., Eiberger, O., Friedl, W., Bäuml, B., Hillenbrand, U., Borst, C., Albu-Schäffer, A., Brunner, B., Hirschmüller, H., Kielhöfer, S., Konietschke, R., Suppa, M., Wimböck, T., Zacharias, F., Hirzinger, G.: A humanoid two-arm system for dexterous manipulation. In: 6th IEEE-RAS International Conference on Humanoid Robots, Genova (2006)Google Scholar
  33. 33.
    Palli, G., Pirozzi, S.: Force sensor based on discrete optoelectronic components and compliant frames. Sensors and Actuators A: Physical 165, 239–249 (2011)CrossRefGoogle Scholar
  34. 34.
    Palli, G., Pirozzi, S.: Miniaturized optical-based force sensors for tendon-driven robots. In: IEEE International Conference on Robotics and Automation, Shanghai (2011)Google Scholar
  35. 35.
    Park, F., Brockett, R.: Kinematic dexterity of robotic mechanisms. International Journal of Robotics Research 13, 1–15 (1994)CrossRefGoogle Scholar
  36. 36.
    Park, J.Y., Chang, P.H., Yang, J.Y.: Task-oriented design of robot kinematics using the grid method. Advanced Robotics 17, 879–907 (2003)CrossRefGoogle Scholar
  37. 37.
    Pettré, J., Laumond, J.P., Siméon, T.: A 2-stage locomotion planner for digital actors. In: Eurographics Symposium on Computer Animation, San Diego, CA (2003)Google Scholar
  38. 38.
    Pin, F.G., Culioli, J.C.: Optimal positioning of combined mobile platform-manipulator systems for material handling tasks. Journal of Intelligent and Robotic Systems 6, 165–182 (1992)CrossRefGoogle Scholar
  39. 39.
    Pitarch, E., Yang, J., Abdel-Malek, K.: Santos hand: A 25 degree-of-freedom model. In: Digital Human Modeling for Design and Engineering Symposium, Iowa City, IA (2005)Google Scholar
  40. 40.
    Pressure-Profile: Fingertps (2010),
  41. 41.
    Ristic, R., Arulampalam, S., Gordon, N.: Beyond the Kalman Filter. Artech House Publishers, Boston (2004)zbMATHGoogle Scholar
  42. 42.
    Roa, M.A., Hertkorn, K., Borst, C., Hirzinger, G.: Reachable independent contact regions for precision grasps. In: IEEE International Conference on Robotics and Automation, Shanghai (2011)Google Scholar
  43. 43.
    Saff, E., Kuijlaars, A.: Distributing many points on the sphere. Mathematical Intelligence 19(1), 5–11 (1997)MathSciNetzbMATHCrossRefGoogle Scholar
  44. 44.
    Sagardia, M., Hulin, T., Preusche, C., Hirzinger, G.: Improvements of the voxmap-pointshell algorithm – Fast generation of haptic data structures. In: 53rd Internationales Wissenschaftliches Kolloquium, Ilmenau (2007)Google Scholar
  45. 45.
    Shannon, C.E.: Communication in the presence of noise. Proceedings of the Institute of Radio Engineers 37(1), 10–21 (1949)MathSciNetGoogle Scholar
  46. 46.
    Siciliano, B., Sciavicco, L., Villani, L., Oriolo, G.: Robotics: Modelling, Planning and Control. Springer, London (2009)Google Scholar
  47. 47.
    Stilman, M.: Task constrained motion planning in robot joint space. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA (2007)Google Scholar
  48. 48.
    Stocco, L., Salcudean, S.E., Sassani, F.: Fast constrained global minimax optimization of robot parameters. Robotica 16, 595–605 (1998)CrossRefGoogle Scholar
  49. 49.
    Sturges, R.: A quantification of machine dexterity applied to an assembly task. International Journal of Robotics Research 9(3), 49–62 (1990)CrossRefGoogle Scholar
  50. 50.
    Townsend, W.: The Barrett hand grasper — Programmably flexible part handling and assembly. Industrial Robot 27(3), 181–188 (2000)MathSciNetCrossRefGoogle Scholar
  51. 51.
    Vaughan, J., Rosenbaum, D., Meulenbroek, R.: Planning reaching and grasping movements: The problem of obstacle avoidance. Motor Control 5(2), 116–135 (2001)Google Scholar
  52. 52.
    Veber, M., Bajd, T., Munih, M.: Assessing joint angles in human hand via optical tracking device and calibrating instrumented glove. Meccanica 42, 451–463 (2007)zbMATHCrossRefGoogle Scholar
  53. 53.
    Yoshikawa, T.: Foundations of Robotics: Analysis and Control. MIT Press, Cambridge (1990)Google Scholar
  54. 54.
    Zacharias, F., Borst, C., Beetz, M., Hirzinger, G.: Positioning mobile manipulators to perform constrained linear trajectories. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Nice (2008)Google Scholar
  55. 55.
    Zacharias, F., Borst, C., Hirzinger, G.: Capturing robot workspace structure: Representing robot capabilities. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA (2007)Google Scholar
  56. 56.
    Zacharias, F., Borst, C., Hirzinger, G.: Object-specific grasp maps for use in planning manipulation actions. In: German Workshop on Robotics, Braunschweig (2009)Google Scholar
  57. 57.
    Zhang, X., Lee, S., Braido, P.: Determining finger segmental centers of rotation in flexion-extension based on surface marker measurement. Journal of Biomechanics 36, 1097–1102 (2003)CrossRefGoogle Scholar
  58. 58.
    Zhu, X., Ding, H., Tso, S.: A pseudodistance function and its applications. IEEE Transactions on Robotics and Automation 20, 344–352 (2004)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Berlin Heidelberg 2012

Authors and Affiliations

  • Christoph Borst
    • 1
  • Franziska Zacharias
    • 1
  • Florian Schmidt
    • 1
  • Daniel Leidner
    • 1
  • Maximo A. Roa
    • 1
  • Katharina Hertkorn
    • 1
  • Gerhard Grunwald
    • 1
  • Pietro Falco
    • 2
  • Ciro Natale
    • 2
  • Emilio Maggio
    • 3
  1. 1.Robotik und Mechatronik Zentrum, DLRWesslingGermany
  2. 2.Dipartimento di Ingegneria dell’InformazioneSeconda Università degli Studi di NapoliAversaItaly
  3. 3.Oxford Metrics GroupOxfordUnited Kingdom

Personalised recommendations