Skip to main content

Human-Robot Collaboration and Dialogue for Fault Recovery on Hierarchical Tasks

Part of the Lecture Notes in Computer Science book series (LNAI,volume 12483)


Robotic systems typically follow a rigid approach to task execution, in which they perform the necessary steps in a specific order, but fail when having to cope with issues that arise during execution. We propose an approach that handles such cases through dialogue and human-robot collaboration. The proposed approach contributes a hierarchical control architecture that 1) autonomously detects and is cognizant of task execution failures, 2) initiates a dialogue with a human helper to obtain assistance, and 3) enables collaborative human-robot task execution through extended dialogue in order to 4) ensure robust execution of hierarchical tasks with complex constraints, such as sequential, non-ordering, and multiple paths of execution. The architecture ensures that the constraints are adhered to throughout the entire task execution, including during failures. The recovery of the architecture from issues during execution is validated by a human-robot team on a building task.


  • Human-robot collaboration
  • Human-robot dialogue
  • Dialogue-based fault recovery
  • Hierarchical planning

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-62056-1_13
  • Chapter length: 13 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   109.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-62056-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   139.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.


  1. Arkin, R.C.: An Behavior-based Robotics, 1st edn. MIT Press, Cambridge (1998)

    Google Scholar 

  2. Beetz, M., et al.: Robotic roommates making pancakes. In: 2011 11th IEEE-RAS International Conference on Humanoid Robots, pp. 529–536, October 2011

    Google Scholar 

  3. Blankenburg, J., et al.: A distributed control architecture for collaborative multi-robot task allocation. In: 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids), pp. 585–592, November 2017

    Google Scholar 

  4. Bradski, G., Kaehler, A.: Learning OpenCV: Computer vision with the OpenCV library. O’Reilly Media, Inc. (2008)

    Google Scholar 

  5. Breazeal, C., Hoffman, G., Lockerd, A.: Teaching and working with robots as a collaboration. In: Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems-Volume 3, pp. 1030–1037. IEEE Computer Society (2004)

    Google Scholar 

  6. Fong, T., Thorpe, C., Baur, C.: Multi-robot remote driving with collaborative control. IEEE Trans. Ind. Electron. 50(4), 699–704 (2003)

    CrossRef  Google Scholar 

  7. Fong, T., Thorpe, C., Baur, C.: Robot, asker of questions. Robot. Autonomous Syst. 42(3), 235–243 (2003)., socially Interactive Robots

  8. Fraser, L., Rekabdar, B., Nicolescu, M., Nicolescu, M., Feil-Seifer, D., Bebis, G.: A compact task representation for hierarchical robot control. In: 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids), pp. 697–704, November 2016.

  9. Gassend, B.: sound\_play ros package.

  10. Hayes, B., Scassellati, B.: Autonomously constructing hierarchical task networks for planning and human-robot collaboration. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 5469–5476. IEEE (2016)

    Google Scholar 

  11. Hayes, B., Shah, J.A.: Improving robot controller transparency through autonomous policy explanation. In: 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 303–312. IEEE (2017)

    Google Scholar 

  12. Hernandez, C., et al.: Team delft’s robot winner of the amazon picking challenge 2016. In: Behnke, S., Sheh, R., Sarıel, S., Lee, D.D. (eds.) RoboCup 2016: Robot World Cup XX, pp. 613–624. Springer International Publishing, Cham (2017)

    CrossRef  Google Scholar 

  13. Huggins-Daines, D., Kumar, M., Chan, A., Black, A.W., Ravishankar, M., Rudnicky, A.I.: Pocketsphinx: a free, real-time continuous speech recognition system for hand-held devices. In: 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, vol. 1, p. I. IEEE (2006)

    Google Scholar 

  14. Knepper, R.A., Tellex, S., Li, A., Roy, N., Rus, D.: Recovering from failure by asking for help. Auton. Robots 39(3), 347–362 (2015).

    CrossRef  Google Scholar 

  15. Mohseni-Kabir, A., Rich, C., Chernova, S., Sidner, C.L., Miller, D.: Interactive hierarchical task learning from a single demonstration. In: Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction, pp. 205–212. ACM (2015)

    Google Scholar 

  16. Quigley, M., et al.: ROS: an open-source Robot Operating System. In: ICRA Workshop on Open Source Software, vol. 3 (2009)

    Google Scholar 

  17. Rybski, P.E., Yoon, K., Stolarz, J., Veloso, M.M.: Interactive robot task training through dialog and demonstration. In: Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction, pp. 49–56. ACM (2007)

    Google Scholar 

  18. Schillinger, P., Kohlbrecher, S., von Stryk, O.: Human-robot collaborative high-level control with application to rescue robotics. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 2796–2802. IEEE (2016)

    Google Scholar 

  19. Sucan, I.A., Chitta, S.: Moveit! (2013).

Download references


This work was supported by the National Science Foundation (IIS-1757929) and by the Office of Naval Research (ONR) award #N00014-16-1-2312.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Janelle Blankenburg .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Blankenburg, J., Zagainova, M., Simmons, S.M., Talavera, G., Nicolescu, M., Feil-Seifer, D. (2020). Human-Robot Collaboration and Dialogue for Fault Recovery on Hierarchical Tasks. In: , et al. Social Robotics. ICSR 2020. Lecture Notes in Computer Science(), vol 12483. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-62055-4

  • Online ISBN: 978-3-030-62056-1

  • eBook Packages: Computer ScienceComputer Science (R0)