International Journal of Social Robotics

, Volume 6, Issue 4, pp 533–553 | Cite as

An Attentional Approach to Human–Robot Interactive Manipulation

  • Xavier Broquère
  • Alberto Finzi
  • Jim Mainprice
  • Silvia Rossi
  • Daniel Sidobre
  • Mariacarla Staffa


Human robot collaborative work requires interactive manipulation and object handover. During the execution of such tasks, the robot should monitor manipulation cues to assess the human intentions and quickly determine the appropriate execution strategies. In this paper, we present a control architecture that combines a supervisory attentional system with a human aware manipulation planner to support effective and safe collaborative manipulation. After detailing the approach, we present experimental results describing the system at work with different manipulation tasks (give, receive, pick, and place).


Human–robot interaction Cognitive control Robot manipulation Attentional system Human aware planning and execution 



The research leading to these results has been supported by the SAPHARI Large-scale integrating project, which has received funding from the European Community’s Seventh Framework Programme (FP7/2007-2013) under grant agreement ICT-287513. The authors are solely responsible for its content. It does not represent the opinion of the European Community and the Community is not responsible for any use that might be made of the information contained therein.


  1. 1.
    Abbeel P, Ng AY (2004) Apprenticeship learning via inverse reinforcement learning. In: Proceedings of the twenty-first international conference on Machine learning, p 1. ACMGoogle Scholar
  2. 2.
    Alili S, Alami R, Montreuil V (2009) A task planner for an autonomous social robot. In: Distributed autonomous robotic systems. Springer, Berlin, pp 335–344Google Scholar
  3. 3.
    Arbib MA (1998) Schema theory. In: The handbook of brain theory and neural networks. MIT Press, Cambridge, pp 830–834Google Scholar
  4. 4.
    Arkin R (1998) Behavior based robotics. MIT Press, CambridgeGoogle Scholar
  5. 5.
    Berchtold S, Glavina B (1994) A scalable optimizer for automatically generated manipulator motions. In: IEEE/RSJ Int. Conf. on Intel. Rob. And Sys. IEEE, Munich, GermanyGoogle Scholar
  6. 6.
    Bounab B, Labed A, Sidobre D (2010) Stochastic optimization-based approach for multifingered grasps synthesis. Robotica 28(07):1021–1032CrossRefGoogle Scholar
  7. 7.
    Breazeal C (2002) Designing sociable robots. MIT Press, CambridgeGoogle Scholar
  8. 8.
    Breazeal C, Kidd CD, Thomaz AL, Hoffman G, Berlin M (2005) Effects of nonverbal communication on efficiency and robustness in human-robot teamwork. In: in IROS-2005. ACM/IEEE, Edmonton, pp 383–388Google Scholar
  9. 9.
    Brooks RA (1991) A robust layered control system for a mobile robot. In: Iyengar SS, Elfes A (eds) Autonomous mobile robots: control, planning, and architecture (vol 2). IEEE Computer Society Press, Los Alamitos, pp 152–161Google Scholar
  10. 10.
    Broquère X, Sidobre D (2010) From motion planning to trajectory control with bounded jerk for service manipulator robots. In: IEEE Int. Conf. Robot. And Autom. IEEE, AnchorageGoogle Scholar
  11. 11.
    Broquère X, Sidobre D, Herrera-Aguilar I (2008) Soft motion trajectory planner for service manipulator robot. In: IEEE/RSJ Int. Conf. on Intel. Rob. And Sys. IEEE, Nice, FranceGoogle Scholar
  12. 12.
    Burattini E, Finzi A, Rossi S, Staffa M (2010) Attentive monitoring strategies in a behavior-based robotic system: an evolutionary approach. In: Proceedings of the 2010 international conference on emerging security technologies, EST ’10. IEEE Computer Society, Washington, pp 153–158Google Scholar
  13. 13.
    Burattini E, Finzi A, Rossi S, Staffa M (2011) Cognitive control in cognitive robotics: attentional executive control. In: Proc. of ICAR-2011. IEEE, Tallin, Estonia, pp 359–364Google Scholar
  14. 14.
    Burattini E, Finzi A, Rossi S, Staffa M (2012) Attentional human-robot interaction in simple manipulation tasks. In: Proc. of HRI-2012, Late-Breaking Reports. ACM/IEEE, BostonGoogle Scholar
  15. 15.
    Burattini E, Rossi S (2008) Periodic adaptive activation of behaviors in robotic system. IJPRAI 22(5):987–999 Special Issue on Brain, Vision and Artificial IntelligenceGoogle Scholar
  16. 16.
    Clodic A, Cao H, Alili S, Montreuil V, Alami R, Chatila R (2009) Shary: a supervision system adapted to human-robot interaction. In: Khatib O, Kumar V, Pappas G (eds) Experimental robotics, springer tracts in advanced robotics, vol 54. Springer, Berlin, pp. 229–238. doi: 10.1007/978-3-642-00196-3_27
  17. 17.
    Cooper R, Shallice T (2000) Contention scheduling and the control of routine activities. Cogn Neuropsychol 17:297–338CrossRefGoogle Scholar
  18. 18.
    Di Nocera D, Finzi A, Rossi S, Staffa M (2012) Attentional action selection using reinforcement learning. In: Ziemke T, Balkenius C, Hallam J (eds) From animals to animats 12–12th international conference on simulation of adaptive behavior, SAB 2012, Lecture Notes in Computer Science, vol 7426. Springer, Berlin, pp 371–380Google Scholar
  19. 19.
    Duguleana M, Barbuceanu FG, Mogan G (2011) Evaluating human-robot interaction during a manipulation experiment conducted in immersive virtual reality. In: Proc. of international conference on virtual and mixed reality: new trends, vol I. Springer, Berlin, pp 164–173Google Scholar
  20. 20.
    Edsinger A, Kemp CC (2007) Human-robot interaction for cooperative manipulation: Handing objects to one another. In: RO-MAN 2007. IEEE, Jeju, Korea, pp 1167–1172Google Scholar
  21. 21.
    Fitts P (1954) The information capacity of the human motor system in controlling the amplitude of movement. J Exp Psychol 47(6):381391CrossRefGoogle Scholar
  22. 22.
    Fleury S, Herrb M, Chatila R (1997) Genom: a tool for the specification and the implementation of operating modules in a distributed robot architecture. In: IEEE/RSJ Int. conf. on intel. rob. snd sys. IEEE, Grenoble, FranceGoogle Scholar
  23. 23.
    Hoffman G, Breazeal C (2007) Cost-based anticipatory action selection for human–robot fluency. IEEE Trans Robot 23(5): 952–961Google Scholar
  24. 24.
    Iengo S, Origlia A, Staffa M, Finzi A (2012) Attentional and emotional regulation in human-robot interaction. In: RO-MAN, pp 1135–1140Google Scholar
  25. 25.
    Itti L, Koch C (2001) Computational modeling of visual attention. Nat Rev Neurosci 2(3):194–203CrossRefGoogle Scholar
  26. 26.
    Jaillet L, Cortés J, Siméon T (2010) Sampling-based path planning on configuration-space costmaps. IEEE Trans Robot 26(4): 635–646Google Scholar
  27. 27.
    Kahneman D (1973) Attention and effort. Prentice-Hall, EnglewoodGoogle Scholar
  28. 28.
    Kaplan F, Hafner VV (2006) The challenges of joint attention. Interact Stud 7(2):135–169. doi: 10.1075/is.7.2.04kap CrossRefGoogle Scholar
  29. 29.
    Lang S, Kleinehagenbrock M, Hohenner S, Fritsch J, Fink GA, Sagerer G (2003) Providing the basis for human-robot-interaction: A multi-modal attention system for a mobile robot. In: Proc. int. conf. on multimodal interfaces. ACM, Vancouver, pp 28–35Google Scholar
  30. 30.
    Mainprice J, Sisbot E, Jaillet L, Cortés J, Siméon T, Alami R (2011) Planning Human-aware motions using a sampling-based costmap planner. In: IEEE int. conf. robot. and autom. IEEE, Shanghai.Google Scholar
  31. 31.
    Marler R, Rahmatalla S, Shanahan M, Abdel-Malek K (2005) A new discomfort function for optimization-based posture prediction. SAE Technical Paper, WarrendaleGoogle Scholar
  32. 32.
    Nagai Y, Hosoda K, Morita A, Asada M (2003) A constructive model for the development of joint attention. Connect Sci 15(4):211–229CrossRefGoogle Scholar
  33. 33.
    Norman D, Shallice T (1986) Attention in action: willed and automatic control of behaviour. Conscious Self-Regulation 4:1–18CrossRefGoogle Scholar
  34. 34.
    Pashler H, Johnston J (1998) Attentional limitations in dual-task performance. In: Pashler H (ed) Attention. Psychology Press, East Essex, pp 155–189Google Scholar
  35. 35.
    Posner M, Snyder C (1975) Attention and cognitive control. In: Information processing and cognition: the loyola symposium. Psychology Pr, Hillsdale, ErlbaumGoogle Scholar
  36. 36.
    Posner M, Snyder C, Davidson B (1980) Attention and the detection of signals. J Exp Psychol Gen 109:160–174CrossRefGoogle Scholar
  37. 37.
    Rossi S, Leone E, Fiore M, Finzi A, Cutugno F, (2013) An extensible architecture for robust multimodal human-robot communication. In: Proc. of IROS, (2013) IEEE. Tokyo, JapanGoogle Scholar
  38. 38.
    Saut JP, Sidobre D (2012) Efficient models for grasp planning with a multi-fingered hand. Robot Auton Syst 60(3):347–357. doi: 10.1016/j.robot.2011.07.019 Autonomous GraspingCrossRefGoogle Scholar
  39. 39.
    Scassellati B (1999) Imitation and mechanisms of joint attention: a developmental structure for building social skills on a humanoid robot. In: Computation for metaphors, analogy and agents, vol 1562. Springer, Berlin, pp 176–195Google Scholar
  40. 40.
    Senders J (1964), The human operator as a monitor and controller of multidegree of freedom systems. IEEE Trans. on Human Factors in, Electronics, HFE-5 pp 2–6Google Scholar
  41. 41.
    Siciliano B (2012) Advanced bimanual manipulation: results from the DEXMART project, vol 80. Springer, Heidelberg. doi: 10.1007/978-3-642-29041-1
  42. 42.
    Sisbot E, Marin-Urias L, Broquère X, Sidobre D, Alami R (2010) Synthesizing robot motions adapted to human presence. Int J Soc Robot 2(3):329–343CrossRefGoogle Scholar
  43. 43.
    Sisbot EA, Alami R (2012) A human-aware manipulation planner. Robot IEEE Trans 28(5):1045–1057CrossRefGoogle Scholar
  44. 44.
    Sisbot EA, Marin-Urias LF, Alami R, Siméon T (2007) Human aware mobile robot motion planner. IEEE Trans Robot 23(5): 874–883CrossRefGoogle Scholar
  45. 45.
    Sisbot EA, Ros R, Alami R (2011) Situation assessment for human-robot interactive object manipulation. In: IEEE RO-MAN. IEEE, IEEE, AtlantaGoogle Scholar
  46. 46.
    Tinbergen N (1951) The study of instinct. Oxford University Press, LondonGoogle Scholar
  47. 47.
    Trafton JG, Cassimatis NL, Bugajska MD, Brock DP, Mintz FE, Schultz AC (2005) Enabling effective human-robot interaction using perspective-taking in robots. IEEE Trans Syst Man Cybern 35:460–470CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2014

Authors and Affiliations

  • Xavier Broquère
    • 1
    • 2
  • Alberto Finzi
    • 4
  • Jim Mainprice
    • 5
  • Silvia Rossi
    • 4
  • Daniel Sidobre
    • 1
    • 3
  • Mariacarla Staffa
    • 4
  1. 1.CNRS, LAASToulouseFrance
  2. 2.Univ de Toulouse, LAASToulouseFrance
  3. 3.Univ de Toulouse, UPS, LAASToulouseFrance
  4. 4.Dipartimento di Ingegneria Elettrica e Tecnologie dell’Informazione (DIETI)Università di degli Studi Napoli Federico IINapoliItaly
  5. 5.Worcester Polytechnic InstituteWorcesterUSA

Personalised recommendations