International Journal of Social Robotics

, Volume 6, Issue 4, pp 593–620 | Cite as

Towards Human-Level Semantics Understanding of Human-Centered Object Manipulation Tasks for HRI: Reasoning About Effect, Ability, Effort and Perspective Taking

  • Amit Kumar Pandey
  • Rachid Alami


In its lifetime, a robot should be able to autonomously understand the semantics of different tasks to effectively perform them in different situations. In this context, it is important to distinguish the meaning (in terms of the desired effect) of a task and the means to achieve that task. Our focus is those tasks in which one agent is required to perform a task for another agent, such as give, show, hide, make-accessible, etc. In this paper, we identify that a high-level human-centered combined reasoning, based on perspective taking, efforts and abilities analyses, is the key to understand semantics of such tasks. By combining these aspects, the robot infers sets of hierarchy of facts, which serve for analyzing the effect of a task. We adapt the explanation based learning approach enabling the task understanding from the very first demonstration and continuous refinement with new demonstrations. We argue that such symbolic level understanding of a task, which is not bound to trajectory, kinematics structure or shape of the robot, facilitates generalization to novel situations as well as ease the transfer of acquired knowledge among heterogeneous robots. Further, the knowledge of tasks at such human understandable level of abstraction will enrich the natural human–robot interaction.


Perspective taking Ability analysis  Effort analysis Emulation learning Effect understanding Social learning Explanation based learning Task semantics 


  1. 1.
    Agostini A, Torras C, Wörgötter F (2011) Integrating task planning and interactive learning for robots to work in human environments. In: Proceedings of the 22nd international conference on artificial intelligence (IJCAI), pp 2386–2391Google Scholar
  2. 2.
    Alili S, Alami R, Montreuil V (2008) A task planner for an autonomous social robot. In: Distributed autonomous robotic systems (DARS), pp 335–344Google Scholar
  3. 3.
    Alili S, Pandey A, Sisbot EA, Alami R (2010) Interleaving symbolic and geometric reasoning for a robotic assistant. In: ICAPS workshop on combining action and motion planning (CAMP)Google Scholar
  4. 4.
    Argall BD, Chernova S, Veloso M, Browning B (2009) A survey of robot learning from demonstration. Robot Auton Syst 57(5):469–483CrossRefGoogle Scholar
  5. 5.
    Awaad I, Kraetzschmar GK, Hertzberg J (2013) Affordance-based reasoning in robot task planning. In: Planning and robotics (PlanRob) workshop at 23rd international conference on automated planning and scheduling (ICAPS)Google Scholar
  6. 6.
    Breazeal C, Berlin M, Brooks AG, Gray J, Thomaz AL (2006) Using perspective taking to learn from ambiguous demonstrations. Robot Auton Syst 54:385–393CrossRefGoogle Scholar
  7. 7.
    Calinon S, Dhalluin F, Caldwell D, Billard A (2009) Handling of multiple constraints and motion alternatives in a robot programming by demonstration framework. In: 9th IEEE-RAS international conference on humanoid robots, humanoids 2009, pp 582–588Google Scholar
  8. 8.
    Cambon S, Alami R, Gravot F (2009) A hybrid approach to intricate motion, manipulation and task planning. Int J Robot Res 28(1):104–126CrossRefGoogle Scholar
  9. 9.
    Cantrell R, Schermerhorn P, Scheutz M (2011) Learning actions from human-robot dialogues. In: IEEE RO-MAN, pp 125–130Google Scholar
  10. 10.
    Carello C, Grosofsky A, Reichel FD, Solomon HY, Turvey M (1989) Visually perceiving what is reachable. Ecol Psychol 1(1):27–54CrossRefGoogle Scholar
  11. 11.
    Carpenter M, Call J (2002) The chemistry of social learning. Dev Sci 5(1):22–24CrossRefGoogle Scholar
  12. 12.
    Cestnik B (1990) Estimating probabilities: a crucial task in machine learning. In: Proceedings of the ninth European conference on artificial intelligence, ECAI, pp 147–149Google Scholar
  13. 13.
    Chao C, Cakmak M, Thomaz A (2011) Towards grounding concepts for transfer in goal learning from demonstration. In: IEEE international conference on development and learning (ICDL) vol 2, pp 1–6Google Scholar
  14. 14.
    Chella A, Dindo H, Infantino I (2006) A cognitive framework for imitation learning. Robot Auton Syst 54(5):403–408CrossRefGoogle Scholar
  15. 15.
    Choi HJ, Mark LS (2004) Scaling affordances for human reach actions. Hum Mov Sci 23(6):785–806CrossRefGoogle Scholar
  16. 16.
    Dantam N, Essa I, Stilman M (2012) Linguistic transfer of human assembly tasks to robots. In: Proceedings of intelligent robots and systems (IROS), pp 237–242Google Scholar
  17. 17.
    de Silva L, Pandey AK, Alami R (2013) An interface for interleaved symbolic-geometric planning and backtracking. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 232–239Google Scholar
  18. 18.
    De Silva L, Gharbi M, Pandey AK, Alami R (2014) A new approach to combined symbolic-geometric backtracking in the context of human–robot interaction. In: IEEE international conference on robotics and automation (ICRA)Google Scholar
  19. 19.
    Dejong G, Mooney R (1986) Explanation-based learning: an alternative view. Mach Learn 1:145–176Google Scholar
  20. 20.
    Dillmann R (2004) Teaching and learning of robot tasks via observation of human performance. Robot Auton Syst 47:109–116CrossRefGoogle Scholar
  21. 21.
    Dragan A, Gordon G, Srinivasa S (2011) Learning from experience in manipulation planning: setting the right goals. In: Proceedings of the international symposium on robotics research (ISRR)Google Scholar
  22. 22.
    Ekvall S, Kragic D (2008) Robot learning from demonstration: a task-level planning approach. Int J Adv Robot Syst 5(3):223–234Google Scholar
  23. 23.
    Flann NS, Dietterich GT (1989) A study of explanation-based methods for inductive learning. Mach Learn 4:187–226CrossRefGoogle Scholar
  24. 24.
    Flavell JH, Everett BA, Croft K, Flavell ER (1981) Young children’s knowledge about visual perception: further evidence for the level 1–level 2 distinction, pp 99–103Google Scholar
  25. 25.
    Flavell JH, Shipstead SG, Croft K (1978) Young children’s knowledge about visual perception: Hiding objects from others. Child Dev 49(4):1208–1211CrossRefGoogle Scholar
  26. 26.
    Furnkranz J, Flach P (2003) An analysis of rule evaluation metrics. In: Proceedings of the 20th international conference on machine learning (ICML). AAAI Press, pp 202–209Google Scholar
  27. 27.
    Gardner DL, Mark LS, Ward JA, Edkins H (2001) How do task characteristics affect the transitions between seated and standing reaches? Ecol Psychol 13(4):245–274CrossRefGoogle Scholar
  28. 28.
    Gribovskaya E, Khansari-Zadeh S, Billard A (2011) Learning non-linear multivariate dynamics of motion in robotic manipulators. Int J Robot Res 30(1):80–117CrossRefGoogle Scholar
  29. 29.
    Hopper LM, Lambeth SP, Schapiro SJ, Whiten A (2008) Observational learning in chimpanzees and children studied through ghostconditions. Proc R Soc B Biol Sci 275(1636):835–840CrossRefGoogle Scholar
  30. 30.
    Johnson M, Demiris Y (2005) Perceptual perspective taking and action recognition. Int J Soc Robot 2:181–199Google Scholar
  31. 31.
    Jkel R, Schmidt-Rohr S, Rhl S, Kasper A, Xue Z, Dillmann R (2012) Learning of planning models for dexterous manipulation based on human demonstrations. Int J Soc Robot 4(4):437–448CrossRefGoogle Scholar
  32. 32.
    Khatib O, Demircan E, Sapio VD, Sentis L, Besier T, Delp S (2009) Robotics-based synthesis of human motion. J Physiol Paris 103:211–219CrossRefGoogle Scholar
  33. 33.
    Kuniyoshi Y, Inaba M, Inoue H (1994) Learning by watching: extracting reusable task knowledge from visual observation of human performance. IEEE Trans Robot Autom 10(6):799–822CrossRefGoogle Scholar
  34. 34.
    Lallee S, Lemaignan S, Lenz A, Melhuish C, Natale L, Skachek S, van Der Zant T, Warneken F, Dominey PF (2010) Towards a platform-independent cooperative human–robot interaction system: I. perception. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 4444–4451Google Scholar
  35. 35.
    Lee KH, Lee J, Thomaz AL, Bobick AF (2009) Effective robot task learning by focusing on task-relevant objects. In: Proceedings of the 2009 IEEE/RSJ international conference on Intelligent robots and systems. IEEE Press, Piscataway, NJ, USA, pp 2551–2556Google Scholar
  36. 36.
    Lemaignan S, Ros R, Mösenlechner L, Alami R, Beetz M (2010) Oro, a knowledge management module for cognitive architectures in robotics. In: Proceedings of IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 3548–3553Google Scholar
  37. 37.
    Lempers JD, Flavell ER, Flavell JH (1977) The development in very young children of tacit knowledge concerning visual perception. Genet Psychol Monogr 95(1):3–53Google Scholar
  38. 38.
    Levas A, Selfridge M (1984) A user-friendly high-level robot teaching system. In: IEEE international conference on robotics and automation. proceedings, vol 1, pp 413–416Google Scholar
  39. 39.
    Lopes M, Melo FS, Montesano L (2007) Affordance-based imitation learning in robots. In: IROS, pp 1015–1021Google Scholar
  40. 40.
    Lunsky LL (1965) Learning and instinct in animals. Arch Intern Med 115(6):757–758CrossRefGoogle Scholar
  41. 41.
    Marin-Urias L, Sisbot E, Pandey A, Tadakuma R, Alami R (2009) Towards shared attention through geometric reasoning for human robot interaction. In: 9th IEEE-RAS international conference on humanoid robots, humanoids 2009, pp 331–336Google Scholar
  42. 42.
    Michael L (2011) Causal learnability. In: Barcelona, International joint conference on artificial intelligence (IJCAI), pp 1014–1020Google Scholar
  43. 43.
    Montesano L, Lopes M, Bernardino A, Santos-Victor J (2007) Modeling affordances using bayesian networks. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 4102–4107Google Scholar
  44. 44.
    Muhlig M, Gienger M, Hellbach S, Steil JJ, Goerick C (2009) Task-level imitation learning using variance-based movement optimization. In: IEEE international conference on robotics and automation, pp 1177–1184Google Scholar
  45. 45.
    Nielsen M (2006) Copying actions and copying outcomes: social learning through the second year. Dev Psychol 42(3):555CrossRefGoogle Scholar
  46. 46.
    Ogawara K, Takamatsu J, Kimura H, Ikeuchi K (2003) Extraction of essential interactions through multiple observations of human demonstrations. IEEE Trans Ind Electron 50(4):667–675CrossRefGoogle Scholar
  47. 47.
    Pandey AK, Alami R (2010) Mightability maps: a perceptual level decisional framework for co-operative and competitive human-robot interaction. In: International conference on intelligent robots and systems (IROS) year 2010, pp 5842–5848Google Scholar
  48. 48.
    Pandey AK, Alami R (2013) Affordance graph: a framework to encode perspective taking and effort based affordances for day-to-day human–robot interaction. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 2180–2187Google Scholar
  49. 49.
    Pandey AK, Alami R (2014) Ingredients and a framework of dexterous manipulation skills for robots in human centered environment and hri. J Robot Soc Jpn 32(4):31–37Google Scholar
  50. 50.
    Pandey AK, Ali M, Alami R (2013) Towards a task-aware proactive sociable robot based on multi-state perspective-taking. Int J Soc Robot 5(2):215–236CrossRefGoogle Scholar
  51. 51.
    Pandey AK, Saut JP, Sidobre D, Alami R (2012) Towards planning human-robot interactive manipulation tasks: task dependent and human oriented autonomous selection of grasp and placement. In: Proceedings of IEEE RAS & EMBS BioRob, pp 1371–1376Google Scholar
  52. 52.
    Pardowitz M, Dillmann R (2007) Towards life-long learning in household robots: the piagetian approach. In: IEEE 6th international conference on development and learning (ICDL), pp 88–93Google Scholar
  53. 53.
    Pasula H, Zettlemoyer L, Kaelbling L (2004) Learning probabilistic relational planning rules. In: International conference on automated planning and scheduling (ICAPS), pp 73–82Google Scholar
  54. 54.
    Piaget J (1945) Play, dreams, and imitation in childhood. Norton, New YorkGoogle Scholar
  55. 55.
    Ros R, Lemaignan S, Sisbot EA, Alami R, Steinwender J, Hamann K, Warneken F (2010) Which one? grounding the referent based on efficient human–robot interaction. In: 19th IEEE international symposium in robot and human interactive communication, pp 570–575Google Scholar
  56. 56.
    Sapio V, Warren J, Khatib O (2006) Predicting reaching postures using a kinematically constrained shoulder model. In: Lennarcic J, Roth B (eds) Advances in robot kinematics. Springer, Berlin, pp 209–218CrossRefGoogle Scholar
  57. 57.
    Saut JP, Sidobre D (2012) Efficient models for grasp planning with a multi-fingered hand. Robot Auton Syst 60(3):347–357CrossRefGoogle Scholar
  58. 58.
    Schmidt-Rohr S, Losch M, Dillmann R (2010) Learning flexible, multi-modal human–robot interaction by observing human–human–interaction. In: IEEE RO-MAN, pp 582–587Google Scholar
  59. 59.
    Simeon T, p. Laumond J, Lamiraux F (2001) Move3d: a generic platform for path planning. In: 4th international symposium on assembly and task planning, pp 25–30Google Scholar
  60. 60.
    Tenorth M, Beetz M (2009) Knowrob- knowledge processing for autonomous personal robots. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 4261–4266Google Scholar
  61. 61.
    Tomasello M (1990) Cultural transmission in the tool use and communicatory signaling of chimpanzees?. Cambridge University Press, Cambridge, MA, pp 274–311Google Scholar
  62. 62.
    Trafton J, Cassimatis N, Bugajska M, Brock D, Mintz F, Schultz A (2005) Enabling effective human–robot interaction using perspective-taking in robots. Syst Man Cybern Part A Syst Hum IEEE Trans 35(4):460–470CrossRefGoogle Scholar
  63. 63.
    Weber BG, Mateas M, Jhala A (2012) Learning from demonstration for goal-driven autonomy. In: AAAIGoogle Scholar
  64. 64.
    Wood DJ (1998) How children think and learn: the social contexts of cognitive development. Blackwell, OxfordGoogle Scholar
  65. 65.
    Wusteman J (1992) Explanation-based learning: a survey. Artif Intell Rev 6:243–262CrossRefGoogle Scholar
  66. 66.
    Ye G, Alterovitz R (2011) Demonstration-guided motion planning. In: Proceedings of the international symposium on robotics research (ISRR)Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2014

Authors and Affiliations

  1. 1.A-Lab, Aldebaran RoboticsParisFrance
  2. 2.CNRS, LAASToulouseFrance
  3. 3.Université de Toulouse, LAASToulouseFrance

Personalised recommendations