Skip to main content
SpringerLink
Log in

From Autonomy to Cooperative Traded Control of Humanoid Manipulation Tasks with Unreliable Communication

Applications to the Valve-turning Task of the DARPA Robotics Challenge and Lessons Learned

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

In this paper, we present our system design, operational procedure, testing process, field results, and lessons learned for the valve-turning task of the DARPA Robotics Challenge (DRC). We present a software framework for cooperative traded control that enables a team of operators to control a remote humanoid robot over an unreliable communication link. Our system, composed of software modules running on-board the robot and on a remote workstation, allows the operators to specify the manipulation task in a straightforward manner. In addition, we have defined an operational procedure for the operators to manage the teleoperation task, designed to improve situation awareness and expedite task completion. Our testing process, consisting of hands-on intensive testing, remote testing, and remote practice runs , demonstrates that our framework is able to perform reliably and is resilient to unreliable network conditions. We analyze our approach, field tests, and experience at the DRC Trials and discuss lessons learned which may be useful for others when designing similar systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Almetwally, I., Mallem, M.: Real-time tele-operation and tele-walking of humanoid robot nao using kinect depth camera. In: 2013 10th IEEE International Conference on Networking, Sensing and Control (ICNSC), pp 463–466 (2013). doi:10.1109/ICNSC.2013.6548783

  2. Alunni, N., Phillips-Grafftin, C., Suay, H.B., Lofaro, D., Berenson, D., Chernova, S., Lindeman, R.W., Oh, P.: Toward a user-guided manipulation framework for high-DOF robots with limited communication. In: 2013 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), pp 1–6 (2013). doi:10.1109/TePRA.2013.6556356

  3. Arkin, R.C., Balch, T.: Aura: principles and practice in review. J. Exp. Theor. Artif. Intell. 9 (2–3), 175–189 (1997)

    Article  Google Scholar 

  4. Balakirsky, S., Carpin, S., Kleiner, A., Lewis, M., Visser, A., Wang, J., Ziparo, V.A.: Towards heterogeneous robot teams for disaster mitigation: results and performance metrics from RoboCup rescue. J. Field Rob. 24(11–12), 943–967 (2007)

    Article  Google Scholar 

  5. Balasubramanian, R., Xu, L., Brook, P.D., Smith, J.R., Matsuoka, Y.: Human-guided grasp measures improve grasp robustness on physical robot. In: 2010 IEEE International Conference on Robotics and Automation (ICRA), pp 2294–2301 (2010). doi:10.1109/ROBOT.2010.5509855

  6. Berenson, D., Srinivasa, S., Kuffner, J.: Task space regions: a framework for pose-constrained manipulation planning. Int. J. Robot. Res. (IJRR) 30(12), 1435–1460 (2011)

    Article  Google Scholar 

  7. Biesiadecki, J., Leger, C., Maimone, M.: Tradeoffs between directed and autonomous driving on the mars exploration rovers. In: Robotics Research, STAR, vol. 26, pp 91–104. Springer, Berlin (2007)

  8. Burridge, R.R., Hambuchen, K.A.: Using prediction to enhance remote robot supervision across time delay. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, 2009, IROS 2009, pp 5628–5634 (2009). doi:10.1109/IROS.2009.5354233

  9. Casper, J., Murphy, R.R.: Human-robot interactions during the robot-assisted urban search and rescue response at the world trade center. IEEE Trans. Syst. Man Cybern. B Cybern. 33(3), 367–385 (2003)

    Article  Google Scholar 

  10. Chitta, S., Jones, E.G., Ciocarlie, M., Hsiao, K.: Perception, planning, and execution for mobile manipulation in unstructured environments. IEEE Robotics and Automation Magazine, Special Issue on Mobile Manipulation (2012)

  11. Cooke, G.W.: U.s. army tank doctrine (2004)

  12. Dang, H., Jun, Y., Oh, P. K., Allen, P.: Planning complex physical tasks for disaster response with a humanoid robot. In: 2013 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), pp 1–6 (2013). doi:10.1109/TePRA.2013.6556365

  13. Dantam, N., Stilman, M.: Robust and efficient communication for real-time multi-process robot software. In: 2012 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids), pp 316–322 (2012). doi:10.1109/HUMANOIDS.2012.6651538

  14. Diankov, R., Kuffner, J.: Openrave: a planning architecture for autonomous robotics. Robotics Institute, Pittsburgh, Tech Rep CMU-RI-TR-08-34. p. 79 (2008)

  15. Diankov, R., Ratliff, N., Ferguson, D., Srinivasa, S., Kuffner, J.: BiSpace planning: concurrent multi-space exploration. In: Robotics: Science and Systems (2008)

  16. Diftler, M.A., Mehling, J. S., Abdallah, M.E., Radford, N.A., Bridgwater, L.B., Sanders, A.M., Askew, R.S., Linn, D.M., Yamokoski, J.D., Permenter, F.A., et al.: Robonaut 2 - the first humanoid robot in space. In: 2011 IEEE International Conference on Robotics and Automation (ICRA), pp 2178–2183 (2011). doi:10.1109/ICRA.2011.5979830

  17. Ferrell, W., Sheridan, T.: Supervisory control of remote manipulation IEEE Spectrum, pp 81–88 (1967)

  18. Goodrich, M.A., Crandall, J.W., Barakova, E.: Teleoperation and beyond for assistive humanoid robots. Reviews of Human Factors and Ergonomics 9(1), 175–226 (2013)

    Article  Google Scholar 

  19. Gossow, D., Leeper, A., Hershberger, D., Ciocarlie, M.: Interactive markers: 3-d user interfaces for ros applications. IEEE Robot. Autom. Mag. 18(4), 14–15 (2011)

    Article  Google Scholar 

  20. Grey, M.X., Dantam, N., Lofaro, D.M., Bobick, A., Egerstedt, M., Oh, P., Stilman, M.: Multi-process control software for HUBO2 Plus robot. In: 2013 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), pp 1–6 (2013). doi:10.1109/TePRA.2013.6556374

  21. Hobbelen, D., de Boer, T., Wisse, M.: System overview of bipedal robots flame and tulip: tailor-made for limit cycle walking. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, 2008. IROS 2008, pp 2486–2491 (2008). doi:10.1109/IROS.2008.4650728

  22. Hornung, A., Bennewitz, M.: Adaptive level-of-detail planning for efficient humanoid navigation. In: 2012 IEEE International Conference on Robotics and Automation (ICRA), pp 997–1002 (2012). doi:10.1109/ICRA.2012.6224898

  23. Hornung, A., Phillips, M., Jones, E.G., Bennewitz, M., Likhachev, M., Chitta, S.: Navigation in three-dimensional cluttered environments for mobile manipulation. In: 2012 IEEE International Conference on Robotics and Automation (ICRA), pp 423–429 (2012). doi:10.1109/ICRA.2012.6225029

  24. Jackson, J.: Microsoft robotics studio: a technical introduction. IEEE Robot. Autom. Mag. 14(4), 82–87 (2007)

    Article  Google Scholar 

  25. Kragic, D., Miller, A.T., Allen, P.K.: Real-time tracking meets online grasp planning. In: Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation, 2001, vol. 3, pp 2460–2465 (2001). doi:10.1109/ROBOT.2001.932992

  26. Lavalle, S.M.: Rapidly-exploring random trees: A new tool for path planning. Tech. rep. (1998)

  27. Lum, M.J.H., Rosen, J., King, H., Friedman, D.C.W., Lendvay, T.S., Wright, A.S., Sinanan, M.N., Hannaford, B.: Teleoperation in surgical robotics – network latency effects on surgical performance. In: Annual International Conference of the IEEE, Engineering in Medicine and Biology Society, 2009. EMBC 2009, pp 6860–6863 (2009). doi:10.1109/IEMBS.2009.5333120

  28. Luo, J., Zhang, Y., Hauser, K., Park, H.A., Paldhe, M., Lee, C.S.G., Grey, M., Stilman, M., Oh, J.H., Lee, J., Kim, I., Oh, P.: Robust ladder-climbing with a humanoid robot with application to the DARPA robotics challenge. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp 2792–2798 (2014). doi:10.1109/ICRA.2014.6907259

  29. Medeiros, A.A.D.: A survey of control architectures for autonomous mobile robots. J. Braz. Comput. Soc. 4(3) (1998)

  30. Miller, A., Allen, P.: Graspit! a versatile simulator for robotic grasping. IEEE Robot. Autom. Mag. 11(4), 110–122 (2004)

    Article  Google Scholar 

  31. Oestges, C., Montenegro-Villacieros, B., Vanhoenacker-Janvier, D.: Modeling propagation into collapsed buildings for radio-localization-based rescue search missions. In: Antennas and Propagation Society International Symposium, 2009. APSURSI ’09, pp 1–4. IEEE (2009)

  32. O’Flaherty, R., Vieira, P., Grey, M.X., Oh, P., Bobick, A., Egerstedt, M., Stilman, M.: Humanoid robot teleoperation for tasks with power tools. In: 2013 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), pp 1–6 (2013). doi:10.1109/TePRA.2013.6556362

  33. Phillips-Grafflin, C.: Unreliable Network Communication Toolkit. https://github.com/WPI-ARC/teleop_toolkit (2013)

  34. Pirjanian, P., Huntsberger, T.L., Trebi-ollennu, A., Aghazarian, H., Das, H., Joshi, S.S., Schenker, P.S.: CAMPOUT: a control architecture for multi-robot planetary outposts. In: Proceedings SPIE Conference Sensor Fusion and Decentralized Control in Robotic Systems III, pp 221–230 (2000)

  35. Pordel, M., Hellstrom, T.: Robotics architecture frameworks, available tools and further requirements. UMINF (2013)

  36. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T.B., Leibs, J., Wheeler, R., Ng, A.Y.: ROS: an open-source robot operating system. In: ICRA Workshop on Open Source Software (2009)

  37. Rasmussen, C., Yuvraj, K., Vallett, R., Sohn, K., Oh, P.: Towards functional labeling of utility vehicle point clouds for humanoid driving. In: 2013 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), pp 1–6 (2013), doi:10.1109/TePRA.2013.6556368

  38. Rusu, R.B., Cousins, S.: 3D is here: point cloud library (PCL). In: 2011 IEEE International Conference on Robotics and Automation (ICRA), pp 1–4 (2011). doi:10.1109/ICRA.2011.5980567

  39. Sakamoto, D., Kanda, T., Ono, T., Ishiguro, H., Hagita, N.: Android as a telecommunication medium with a human-like presence. In: 2007 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp 193–200 (2007)

  40. Sarabia, M., Ros, R., Demiris, Y.: Towards an open-source social middleware for humanoid robots. In: Humanoids (2011)

  41. Senin, P.: Dynamic time warping algorithm review. Information and Computer Science Department University of Hawaii at Manoa Honolulu, USA (2008)

  42. Shannon, C.E.: A Mathematical Theory of Communication. Bell System Technical Journal 27, 379–423, 623–656 (1948)

    Article  MathSciNet  MATH  Google Scholar 

  43. Sheridan, T.B.: Telerobotics, Automation and Human Supervisory Control. MIT Press, Cambridge (1992)

    Google Scholar 

  44. Srinivasa, S., Ferguson, D., Helfrich, C., Berenson, D., Collet, A., Diankov, R., Gallagher, G., Hollinger, G., Kuffner, J.: VandeWeghe M. HERB: a home exploring robotic butler. Autonomous Robots (2009)

  45. Stilman, M.: Task constrained motion planning in robot joint space. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, 2007. IROS 2007, pp 3074–3081 (2007). doi:10.1109/IROS.2007.4399305

  46. Stulp, F., Fedrizzi, A., Mösenlechner, L., Beetz, M.: Learning and reasoning with action-related places for robust mobile manipulation. J. Artif. Intell. Res. 1, 1–42 (2012)

    MathSciNet  MATH  Google Scholar 

  47. Tadokoro, S.: Special project on development of advanced robots for disaster response (DDT project). In: IEEE Workshop on Advanced Robotics and its Social Impacts (2005)

  48. (TTO) DRCTTO: Darpa robotics challenge. Accessed 16 Jan 2013 (2012)

  49. Yakey, J., LaValle, S.M., Kavraki, L.E.: Randomized path planning for linkages with closed kinematics chains. IEEE Trans. Robot. Autom. 17(6), 951–959 (2001)

    Article  Google Scholar 

  50. Yanco, H.A., Drury, J.L., Scholtz, J.: Beyond usability evaluation: analysis of human-robot interaction at a major robotics competition. Hum. Comput. Interact. 19(1–2), 117–149 (2004)

    Article  Google Scholar 

  51. Zacharias, F., Borst, C., Beetz, M., Hirzinger, G.: Positioning mobile manipulators to perform constrained linear trajectories. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, 2008. IROS 2008, pp 2578–2584 (2008). doi:10.1109/IROS.2008.4650617

  52. Zacharias, F., Sepp, W., Borst, C., Hirzinger, G.: Using a model of the reachable workspace to position mobile manipulators for 3-d trajectories. In: 9th IEEE-RAS International Conference on Humanoid Robots, 2009. Humanoids 2009, pp 55–61 (2009). doi:10.1109/ICHR.2009.5379601

  53. Zheng, Y.F., Wang, H., Li, S., Liu, Y., Orin, D., Sohn, K., Jun, Y., Oh, P.: Humanoid robots walking on grass, sands and rocks. In: 2013 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), pp 1–6 (2013). doi:10.1109/TePRA.2013.6556367

  54. Zucker, M., Jun, Y., Killen, B., Kim, T.-G., Oh, P.: Continuous trajectory optimization for autonomous humanoid door opening. In: 2013 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), pp 1–5 (2013). doi:10.1109/TePRA.2013.6556358

Download references

Author information

Authors and Affiliations

  1. Worcester Polytechnic Institute, Worcester, MA, USA

    Calder Phillips-Grafflin, Halit Bener Suay, Jim Mainprice, Nicholas Alunni, Dmitry Berenson, Sonia Chernova & Robert W. Lindeman

  2. Drexel University, Philadelphia, USA

    Daniel Lofaro & Paul Oh

Authors
  1. Calder Phillips-Grafflin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Halit Bener Suay
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jim Mainprice
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nicholas Alunni
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Daniel Lofaro
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Dmitry Berenson
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Sonia Chernova
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Robert W. Lindeman
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Paul Oh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Calder Phillips-Grafflin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Phillips-Grafflin, C., Suay, H.B., Mainprice, J. et al. From Autonomy to Cooperative Traded Control of Humanoid Manipulation Tasks with Unreliable Communication. J Intell Robot Syst 82, 341–361 (2016). https://doi.org/10.1007/s10846-015-0256-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-015-0256-5

Keywords

Search

Navigation