From Autonomy to Cooperative Traded Control of Humanoid Manipulation Tasks with Unreliable Communication

Applications to the Valve-turning Task of the DARPA Robotics Challenge and Lessons Learned
  • Calder Phillips-Grafflin
  • Halit Bener Suay
  • Jim Mainprice
  • Nicholas Alunni
  • Daniel Lofaro
  • Dmitry Berenson
  • Sonia Chernova
  • Robert W. Lindeman
  • Paul Oh


In this paper, we present our system design, operational procedure, testing process, field results, and lessons learned for the valve-turning task of the DARPA Robotics Challenge (DRC). We present a software framework for cooperative traded control that enables a team of operators to control a remote humanoid robot over an unreliable communication link. Our system, composed of software modules running on-board the robot and on a remote workstation, allows the operators to specify the manipulation task in a straightforward manner. In addition, we have defined an operational procedure for the operators to manage the teleoperation task, designed to improve situation awareness and expedite task completion. Our testing process, consisting of hands-on intensive testing, remote testing, and remote practice runs , demonstrates that our framework is able to perform reliably and is resilient to unreliable network conditions. We analyze our approach, field tests, and experience at the DRC Trials and discuss lessons learned which may be useful for others when designing similar systems.


Humanoid robotics Manipulation Teleoperation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Almetwally, I., Mallem, M.: Real-time tele-operation and tele-walking of humanoid robot nao using kinect depth camera. In: 2013 10th IEEE International Conference on Networking, Sensing and Control (ICNSC), pp 463–466 (2013). doi: 10.1109/ICNSC.2013.6548783
  2. 2.
    Alunni, N., Phillips-Grafftin, C., Suay, H.B., Lofaro, D., Berenson, D., Chernova, S., Lindeman, R.W., Oh, P.: Toward a user-guided manipulation framework for high-DOF robots with limited communication. In: 2013 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), pp 1–6 (2013). doi: 10.1109/TePRA.2013.6556356
  3. 3.
    Arkin, R.C., Balch, T.: Aura: principles and practice in review. J. Exp. Theor. Artif. Intell. 9 (2–3), 175–189 (1997)CrossRefGoogle Scholar
  4. 4.
    Balakirsky, S., Carpin, S., Kleiner, A., Lewis, M., Visser, A., Wang, J., Ziparo, V.A.: Towards heterogeneous robot teams for disaster mitigation: results and performance metrics from RoboCup rescue. J. Field Rob. 24(11–12), 943–967 (2007)CrossRefGoogle Scholar
  5. 5.
    Balasubramanian, R., Xu, L., Brook, P.D., Smith, J.R., Matsuoka, Y.: Human-guided grasp measures improve grasp robustness on physical robot. In: 2010 IEEE International Conference on Robotics and Automation (ICRA), pp 2294–2301 (2010). doi: 10.1109/ROBOT.2010.5509855
  6. 6.
    Berenson, D., Srinivasa, S., Kuffner, J.: Task space regions: a framework for pose-constrained manipulation planning. Int. J. Robot. Res. (IJRR) 30(12), 1435–1460 (2011)CrossRefGoogle Scholar
  7. 7.
    Biesiadecki, J., Leger, C., Maimone, M.: Tradeoffs between directed and autonomous driving on the mars exploration rovers. In: Robotics Research, STAR, vol. 26, pp 91–104. Springer, Berlin (2007)Google Scholar
  8. 8.
    Burridge, R.R., Hambuchen, K.A.: Using prediction to enhance remote robot supervision across time delay. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, 2009, IROS 2009, pp 5628–5634 (2009). doi: 10.1109/IROS.2009.5354233
  9. 9.
    Casper, J., Murphy, R.R.: Human-robot interactions during the robot-assisted urban search and rescue response at the world trade center. IEEE Trans. Syst. Man Cybern. B Cybern. 33(3), 367–385 (2003)CrossRefGoogle Scholar
  10. 10.
    Chitta, S., Jones, E.G., Ciocarlie, M., Hsiao, K.: Perception, planning, and execution for mobile manipulation in unstructured environments. IEEE Robotics and Automation Magazine, Special Issue on Mobile Manipulation (2012)Google Scholar
  11. 11.
    Cooke, G.W.: U.s. army tank doctrine (2004)Google Scholar
  12. 12.
    Dang, H., Jun, Y., Oh, P. K., Allen, P.: Planning complex physical tasks for disaster response with a humanoid robot. In: 2013 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), pp 1–6 (2013). doi: 10.1109/TePRA.2013.6556365
  13. 13.
    Dantam, N., Stilman, M.: Robust and efficient communication for real-time multi-process robot software. In: 2012 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids), pp 316–322 (2012). doi: 10.1109/HUMANOIDS.2012.6651538
  14. 14.
    Diankov, R., Kuffner, J.: Openrave: a planning architecture for autonomous robotics. Robotics Institute, Pittsburgh, Tech Rep CMU-RI-TR-08-34. p. 79 (2008)Google Scholar
  15. 15.
    Diankov, R., Ratliff, N., Ferguson, D., Srinivasa, S., Kuffner, J.: BiSpace planning: concurrent multi-space exploration. In: Robotics: Science and Systems (2008)Google Scholar
  16. 16.
    Diftler, M.A., Mehling, J. S., Abdallah, M.E., Radford, N.A., Bridgwater, L.B., Sanders, A.M., Askew, R.S., Linn, D.M., Yamokoski, J.D., Permenter, F.A., et al.: Robonaut 2 - the first humanoid robot in space. In: 2011 IEEE International Conference on Robotics and Automation (ICRA), pp 2178–2183 (2011). doi: 10.1109/ICRA.2011.5979830
  17. 17.
    Ferrell, W., Sheridan, T.: Supervisory control of remote manipulation IEEE Spectrum, pp 81–88 (1967)Google Scholar
  18. 18.
    Goodrich, M.A., Crandall, J.W., Barakova, E.: Teleoperation and beyond for assistive humanoid robots. Reviews of Human Factors and Ergonomics 9(1), 175–226 (2013)CrossRefGoogle Scholar
  19. 19.
    Gossow, D., Leeper, A., Hershberger, D., Ciocarlie, M.: Interactive markers: 3-d user interfaces for ros applications. IEEE Robot. Autom. Mag. 18(4), 14–15 (2011)CrossRefGoogle Scholar
  20. 20.
    Grey, M.X., Dantam, N., Lofaro, D.M., Bobick, A., Egerstedt, M., Oh, P., Stilman, M.: Multi-process control software for HUBO2 Plus robot. In: 2013 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), pp 1–6 (2013). doi: 10.1109/TePRA.2013.6556374
  21. 21.
    Hobbelen, D., de Boer, T., Wisse, M.: System overview of bipedal robots flame and tulip: tailor-made for limit cycle walking. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, 2008. IROS 2008, pp 2486–2491 (2008). doi: 10.1109/IROS.2008.4650728
  22. 22.
    Hornung, A., Bennewitz, M.: Adaptive level-of-detail planning for efficient humanoid navigation. In: 2012 IEEE International Conference on Robotics and Automation (ICRA), pp 997–1002 (2012). doi: 10.1109/ICRA.2012.6224898
  23. 23.
    Hornung, A., Phillips, M., Jones, E.G., Bennewitz, M., Likhachev, M., Chitta, S.: Navigation in three-dimensional cluttered environments for mobile manipulation. In: 2012 IEEE International Conference on Robotics and Automation (ICRA), pp 423–429 (2012). doi: 10.1109/ICRA.2012.6225029
  24. 24.
    Jackson, J.: Microsoft robotics studio: a technical introduction. IEEE Robot. Autom. Mag. 14(4), 82–87 (2007)CrossRefGoogle Scholar
  25. 25.
    Kragic, D., Miller, A.T., Allen, P.K.: Real-time tracking meets online grasp planning. In: Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation, 2001, vol. 3, pp 2460–2465 (2001). doi: 10.1109/ROBOT.2001.932992
  26. 26.
    Lavalle, S.M.: Rapidly-exploring random trees: A new tool for path planning. Tech. rep. (1998)Google Scholar
  27. 27.
    Lum, M.J.H., Rosen, J., King, H., Friedman, D.C.W., Lendvay, T.S., Wright, A.S., Sinanan, M.N., Hannaford, B.: Teleoperation in surgical robotics – network latency effects on surgical performance. In: Annual International Conference of the IEEE, Engineering in Medicine and Biology Society, 2009. EMBC 2009, pp 6860–6863 (2009). doi: 10.1109/IEMBS.2009.5333120
  28. 28.
    Luo, J., Zhang, Y., Hauser, K., Park, H.A., Paldhe, M., Lee, C.S.G., Grey, M., Stilman, M., Oh, J.H., Lee, J., Kim, I., Oh, P.: Robust ladder-climbing with a humanoid robot with application to the DARPA robotics challenge. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp 2792–2798 (2014). doi: 10.1109/ICRA.2014.6907259
  29. 29.
    Medeiros, A.A.D.: A survey of control architectures for autonomous mobile robots. J. Braz. Comput. Soc. 4(3) (1998)Google Scholar
  30. 30.
    Miller, A., Allen, P.: Graspit! a versatile simulator for robotic grasping. IEEE Robot. Autom. Mag. 11(4), 110–122 (2004)CrossRefGoogle Scholar
  31. 31.
    Oestges, C., Montenegro-Villacieros, B., Vanhoenacker-Janvier, D.: Modeling propagation into collapsed buildings for radio-localization-based rescue search missions. In: Antennas and Propagation Society International Symposium, 2009. APSURSI ’09, pp 1–4. IEEE (2009)Google Scholar
  32. 32.
    O’Flaherty, R., Vieira, P., Grey, M.X., Oh, P., Bobick, A., Egerstedt, M., Stilman, M.: Humanoid robot teleoperation for tasks with power tools. In: 2013 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), pp 1–6 (2013). doi: 10.1109/TePRA.2013.6556362
  33. 33.
    Phillips-Grafflin, C.: Unreliable Network Communication Toolkit. (2013)
  34. 34.
    Pirjanian, P., Huntsberger, T.L., Trebi-ollennu, A., Aghazarian, H., Das, H., Joshi, S.S., Schenker, P.S.: CAMPOUT: a control architecture for multi-robot planetary outposts. In: Proceedings SPIE Conference Sensor Fusion and Decentralized Control in Robotic Systems III, pp 221–230 (2000)Google Scholar
  35. 35.
    Pordel, M., Hellstrom, T.: Robotics architecture frameworks, available tools and further requirements. UMINF (2013)Google Scholar
  36. 36.
    Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T.B., Leibs, J., Wheeler, R., Ng, A.Y.: ROS: an open-source robot operating system. In: ICRA Workshop on Open Source Software (2009)Google Scholar
  37. 37.
    Rasmussen, C., Yuvraj, K., Vallett, R., Sohn, K., Oh, P.: Towards functional labeling of utility vehicle point clouds for humanoid driving. In: 2013 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), pp 1–6 (2013), doi: 10.1109/TePRA.2013.6556368
  38. 38.
    Rusu, R.B., Cousins, S.: 3D is here: point cloud library (PCL). In: 2011 IEEE International Conference on Robotics and Automation (ICRA), pp 1–4 (2011). doi: 10.1109/ICRA.2011.5980567
  39. 39.
    Sakamoto, D., Kanda, T., Ono, T., Ishiguro, H., Hagita, N.: Android as a telecommunication medium with a human-like presence. In: 2007 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp 193–200 (2007)Google Scholar
  40. 40.
    Sarabia, M., Ros, R., Demiris, Y.: Towards an open-source social middleware for humanoid robots. In: Humanoids (2011)Google Scholar
  41. 41.
    Senin, P.: Dynamic time warping algorithm review. Information and Computer Science Department University of Hawaii at Manoa Honolulu, USA (2008)Google Scholar
  42. 42.
    Shannon, C.E.: A Mathematical Theory of Communication. Bell System Technical Journal 27, 379–423, 623–656 (1948)MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Sheridan, T.B.: Telerobotics, Automation and Human Supervisory Control. MIT Press, Cambridge (1992)Google Scholar
  44. 44.
    Srinivasa, S., Ferguson, D., Helfrich, C., Berenson, D., Collet, A., Diankov, R., Gallagher, G., Hollinger, G., Kuffner, J.: VandeWeghe M. HERB: a home exploring robotic butler. Autonomous Robots (2009)Google Scholar
  45. 45.
    Stilman, M.: Task constrained motion planning in robot joint space. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, 2007. IROS 2007, pp 3074–3081 (2007). doi: 10.1109/IROS.2007.4399305
  46. 46.
    Stulp, F., Fedrizzi, A., Mösenlechner, L., Beetz, M.: Learning and reasoning with action-related places for robust mobile manipulation. J. Artif. Intell. Res. 1, 1–42 (2012)MathSciNetzbMATHGoogle Scholar
  47. 47.
    Tadokoro, S.: Special project on development of advanced robots for disaster response (DDT project). In: IEEE Workshop on Advanced Robotics and its Social Impacts (2005)Google Scholar
  48. 48.
    (TTO) DRCTTO: Darpa robotics challenge. Accessed 16 Jan 2013 (2012)Google Scholar
  49. 49.
    Yakey, J., LaValle, S.M., Kavraki, L.E.: Randomized path planning for linkages with closed kinematics chains. IEEE Trans. Robot. Autom. 17(6), 951–959 (2001)CrossRefGoogle Scholar
  50. 50.
    Yanco, H.A., Drury, J.L., Scholtz, J.: Beyond usability evaluation: analysis of human-robot interaction at a major robotics competition. Hum. Comput. Interact. 19(1–2), 117–149 (2004)CrossRefGoogle Scholar
  51. 51.
    Zacharias, F., Borst, C., Beetz, M., Hirzinger, G.: Positioning mobile manipulators to perform constrained linear trajectories. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, 2008. IROS 2008, pp 2578–2584 (2008). doi: 10.1109/IROS.2008.4650617
  52. 52.
    Zacharias, F., Sepp, W., Borst, C., Hirzinger, G.: Using a model of the reachable workspace to position mobile manipulators for 3-d trajectories. In: 9th IEEE-RAS International Conference on Humanoid Robots, 2009. Humanoids 2009, pp 55–61 (2009). doi: 10.1109/ICHR.2009.5379601
  53. 53.
    Zheng, Y.F., Wang, H., Li, S., Liu, Y., Orin, D., Sohn, K., Jun, Y., Oh, P.: Humanoid robots walking on grass, sands and rocks. In: 2013 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), pp 1–6 (2013). doi: 10.1109/TePRA.2013.6556367
  54. 54.
    Zucker, M., Jun, Y., Killen, B., Kim, T.-G., Oh, P.: Continuous trajectory optimization for autonomous humanoid door opening. In: 2013 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), pp 1–5 (2013). doi: 10.1109/TePRA.2013.6556358

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  • Calder Phillips-Grafflin
    • 1
  • Halit Bener Suay
    • 1
  • Jim Mainprice
    • 1
  • Nicholas Alunni
    • 1
  • Daniel Lofaro
    • 2
  • Dmitry Berenson
    • 1
  • Sonia Chernova
    • 1
  • Robert W. Lindeman
    • 1
  • Paul Oh
    • 2
  1. 1.Worcester Polytechnic InstituteWorcesterUSA
  2. 2.Drexel UniversityPhiladelphiaUSA

Personalised recommendations