Advertisement

Director: A User Interface Designed for Robot Operation with Shared Autonomy

  • Pat Marion
  • Maurice Fallon
  • Robin Deits
  • Andrés Valenzuela
  • Claudia Pérez D’Arpino
  • Greg Izatt
  • Lucas Manuelli
  • Matt Antone
  • Hongkai Dai
  • Twan Koolen
  • John Carter
  • Scott Kuindersma
  • Russ Tedrake
Chapter
Part of the Springer Tracts in Advanced Robotics book series (STAR, volume 121)

Abstract

Operating a high degree of freedom mobile manipulator, such as a humanoid, in a field scenario requires constant situational awareness, capable perception modules, and effective mechanisms for interactive motion planning and control. A well-designed operator interface presents the operator with enough context to quickly carry out a mission and the flexibility to handle unforeseen operating scenarios robustly. By contrast, an unintuitive user interface can increase the risk of catastrophic operator error by overwhelming the user with unnecessary information. With these principles in mind, we present the philosophy and design decisions behind Director—the open-source user interface developed by Team MIT to pilot the Atlas robot in the DARPA Robotics Challenge (DRC). At the heart of Director is an integrated task execution system that specifies sequences of actions needed to achieve a substantive task, such as drilling a wall or climbing a staircase. These task sequences, developed a priori, make online queries to automated perception and planning algorithms with outputs that can be reviewed by the operator and executed by our whole-body controller. Our use of Director at the DRC resulted in efficient high-level task operation while being fully competitive with approaches focusing on teleoperation by highly-trained operators. We discuss the primary interface elements that comprise the Director and provide analysis of its successful use at the DRC.

Notes

Acknowledgements

We gratefully acknowledge the support of the Defense Advanced Research Projects Agency via Air Force Research Laboratory award FA8750-12-1-0321, and the Office of Naval Research via award N00014-12-1-0071. We are also grateful to the team’s many supporters both inside and outside MIT (listed at http://drc.mit.edu), including our families and friends. We are also grateful to Boston Dynamics, Carnegie Robotics, the Open Source Robotics Foundation, Robotiq, iRobot Corporation, and Sandia National Laboratories for their support during the DRC. Included photos from the DRC Finals are credit to Jason Dorfman of MIT CSAIL. Finally, we acknowledge our late colleague, advisor, and friend, Seth Teller, whose leadership and ideas contributed immensely to this work.

References

  1. Bagnell, J. A., Cavalcanti, F., Cui, L., Galluzzo, T., Hebert, M., Kazemi, M., Klingensmith, M., et al. (2012). An integrated system for autonomous robotics manipulation. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 2955–2962). IEEE.Google Scholar
  2. Bohren, J., Rusu, R. B., Jones, E. G., Marder-Eppstein, E., Pantofaru, C., Wise, M., et al. (2011). Towards autonomous robotic butlers: Lessons learned with the PR2. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 5568–5575). IEEE.Google Scholar
  3. Burstein, M. H., Beranek, B., Inc, N., & Mcdermott, D. V. (1996). Issues in the development of human-computer mixed-initiative planning. In Cognitive technology (pp. 285–303). Elsevier.Google Scholar
  4. Conway, L., Volz, R. A., & Walker, M. W. (1990). Teleautonomous systems: Projecting and coordinating intelligent action at a distance. IEEE Transaction on Robotics, 6(2), 146–158.CrossRefGoogle Scholar
  5. Deits, R., & Tedrake, R. (2014). Footstep planning on uneven terrain with mixed-integer convex optimization. In IEEE/RSJ International Conference on Humanoid Robots (pp. 279–286). IEEE.Google Scholar
  6. Diankov, R. (2010). Automated construction of robotic manipulation programs. Ph.D thesis, Robotics Institute, Carnegie Mellon University.Google Scholar
  7. Dorais, G., Bonasso, R. P., Kortenkamp, D., Pell, B., & Schreckenghost, D. (1999). Adjustable autonomy for human-centered autonomous systems. In Working Notes of the Sixteenth International Joint Conference on Artificial Intelligence Workshop on Adjustable Autonomy Systems (pp. 16–35).Google Scholar
  8. Dragan, A., & Srinivasa, S. (2012). Formalizing assistive teleoperation. Science and Systems (RSS): In Robotics.CrossRefGoogle Scholar
  9. Dragan, A. D., & Srinivasa, S. S. (2013). A policy-blending formalism for shared control. International Journal of Robotics Research, 32(7), 790–805.CrossRefGoogle Scholar
  10. Enes, A. R. (2010). Shared control of hydraulic manipulators to decrease cycle time. Ph.D thesis, Georgia Institute of Technology.Google Scholar
  11. Fallon, M. F., Antone, M., Roy, N., & Teller, S. (2014). Drift-free humanoid state estimation fusing kinematic, inertial and lidar sensing. In IEEE/RSJ International Conference on Humanoid Robots, Madrid, Spain.Google Scholar
  12. Fallon, M., Kuindersma, S., Karumanchi, S., Antone, M., Schneider, T., Dai, H., et al. (2015a). An architecture for online affordance-based perception and whole-body planning. Journal of Field Robotics, 32(2), 229–254.CrossRefGoogle Scholar
  13. Fallon, M. F., Marion, P., Deits, R., Whelan, T., Antone, M., McDonald, J., et al. (2015b). Continuous humanoid locomotion over uneven terrain using stereo fusion. In IEEE/RSJ International Conference on Humanoid Robots, Seoul, Korea. IEEE.Google Scholar
  14. Finzi, A., & Orlandini, A. (2005). Human-robot interaction through mixed-initiative planning for rescue and search rovers. In AI* IA 2005: Advances in artificial intelligence (pp. 483–494). Springer.Google Scholar
  15. Gibson, J. J. (1977). The theory of affordances. In R. Shaw & J. Bransford (Eds.), Perceiving, acting, and knowing. Wiley.Google Scholar
  16. Hart, S., Dinh, P., & Hambuchen, K. (2015). The affordance template ROS package for robot task programming. In 2015 IEEE International Conference on Robotics and Automation (ICRA) (pp. 6227–6234).Google Scholar
  17. Hart, S., Dinh, P., Yamokoski, J. D., Wightman, B., & Radford, N. (2014). Robot task commander: A framework and IDE for robot application development. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2014) (pp. 1547–1554). IEEE.Google Scholar
  18. He, K., Lahijanian, M., Kavraki, L. E., & Vardi, M. Y. (2015). Towards manipulation planning with temporal logic specifications. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 346–352). IEEE.Google Scholar
  19. Huang, A., Olson, E., & Moore, D. (2010). LCM: Lightweight communications and marshalling. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Taipei, Taiwan.Google Scholar
  20. Jain, S., Farshchiansadegh, A., Broad, A., Abdollahi, F., Mussa-Ivaldi, F., & Argall, B. (2015). Assistive robotic manipulation through shared autonomy and a body-machine interface. In 2015 IEEE International Conference on Rehabilitation Robotics (ICORR) (pp. 526–531).Google Scholar
  21. Kaelbling, L. P., & Lozano-Pérez, T. (2011). Hierarchical task and motion planning in the now. In 2011 IEEE International Conference on Robotics and Automation (ICRA) (pp. 1470–1477). IEEE.Google Scholar
  22. Kress-Gazit, H., Fainekos, G. E., & Pappas, G. J. (2007). Where’s Waldo? Sensor-based temporal logic motion planning. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 3116–3121). IEEE.Google Scholar
  23. Kuindersma, S., Deits, R., Fallon, M. F., Valenzuela, A., Dai, H., Permenter, F., et al. (2016). Optimization-based locomotion planning, estimation, and control design for atlas. Autonomous Robots, 40, 429–455.CrossRefGoogle Scholar
  24. Marion, P., Kwitt, R., Davis, B., & Gschwandtner, M. (2012). PCL and ParaView—Connecting the dots. In 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (pp. 80–85). IEEE.Google Scholar
  25. Martin, C. E., Macfadzean, R. H., & Barber, K. S. (1996). Supporting dynamic adaptive autonomy for agent-based systems. In Proceedings of the Artificial Intelligence and Manufacturing Research Planning Workshop (pp. 112–120).Google Scholar
  26. Muelling, K., Venkatraman, A., Valois, J.-S., Downey, J., Weiss, J., Javdani, S., et al. (2015). Autonomy infused teleoperation with application to BCI manipulation. In Robotics Science and Systems (RSS).Google Scholar
  27. Murphy, R. (2004). Human-robot interaction in rescue robotics. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 34(2), 138–153.CrossRefGoogle Scholar
  28. Nehaniv, C. L., & Dautenhahn, K. (2002). The correspondence problem. In Imitation in animals and artifacts (pp. 41–61). Cambridge, MA, USA: MIT Press.Google Scholar
  29. O’Brien, B., Stump, E., & Pierce, C. (2010). Effects of increasing autonomy on tele-operation performance. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 1792–1798).Google Scholar
  30. Rusu, R. B., & Cousins, S. (2011). 3D is here: Point Cloud Library (PCL). In IEEE International Conference on Robotics and Automation (ICRA) (pp. 1–4). IEEE.Google Scholar
  31. Rusu, R. B., Şucan, I. A., Gerkey, B., Chitta, S., Beetz, M., & Kavraki, L. E. (2009). Real-time perception-guided motion planning for a personal robot. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 4245–4252). IEEE.Google Scholar
  32. Schroeder, W. J., Lorensen, B., & Martin, K. (2008). The visualization toolkit: An object-oriented approach to 3D graphics (4th ed.). Kitware.Google Scholar
  33. Sellner, B., Heger, F. W., Hiatt, L. M., Simmons, R., & Singh, S. (2006). Coordinated multiagent teams and sliding autonomy for large-scale assembly. Proceedings of the IEEE, 94(7), 1425–1444.CrossRefGoogle Scholar
  34. Sentis, L., Park, J., & Khatib, O. (2010). Compliant control of multicontact and center-of-mass behaviors in humanoid robots. IEEE Transaction on Robotics, 26(3), 483–501.CrossRefGoogle Scholar
  35. Sheridan, T. B. (1992). Telerobotics, automation, and human supervisory control. Cambridge, MA, USA: MIT Press.Google Scholar
  36. Srivastava, S., Fang, E., Riano, L., Chitnis, R., Russell, S., & Abbeel, P. (2014). Combined task and motion planning through an extensible planner-independent interface layer. In 2014 IEEE International Conference on Robotics and Automation (ICRA) (pp. 639–646). IEEE.Google Scholar
  37. Stentz, A., Herman, H., Kelly, A., Meyhofer, E., Haynes, G. C., Stager, D., et al. (2015). CHIMP, the CMU highly intelligent mobile platform. Journal of Field Robotics, 32(2), 209–228.CrossRefGoogle Scholar
  38. Tambe, M., Scerri, P., & Pynadath, D. V. (2002). Adjustable autonomy for the real world. Journal of Artificial Intelligence Research, 17(1), 171–228.MathSciNetzbMATHGoogle Scholar
  39. Tedrake, R. (2014). Drake: A planning, control, and analysis toolbox for nonlinear dynamical systems. http://drake.mit.edu.
  40. Tedrake, R., Fallon, M., Karumanchi, S., Kuindersma, S., Antone, M., Schneider, T., et al. (2014). A summary of Team MIT’s approach to the virtual robotics challenge. IEEE International Conference on Robotics and Automation (ICRA) (pp. 2087–2087). Hong Kong: China.Google Scholar
  41. Tedrake, R., Kuindersma, S., Deits, R., & Miura, K. (2015). A closed-form solution for real-time ZMP gait generation and feedback stabilization. In IEEE/RSJ International Conference on Humanoid Robots, Seoul, Korea.Google Scholar
  42. Wang, A., Ramos, J., Mayo, J., Ubellacker, W., Cheung, J., & Kim, S. (2015). The HERMES humanoid system: A platform for full-body teleoperation with balance feedback. In IEEE/RSJ International Conference on Humanoid Robots, Seoul, Korea.Google Scholar
  43. Wolfe, J., Marthi, B., & Russell, S. J. (2010). Combined task and motion planning for mobile manipulation. In ICAPS (pp. 254–258).Google Scholar
  44. Yamauchi, B. M. (2004). Packbot: A versatile platform for military robotics. In Proceedings of SPIE 5422. Unmanned Ground Vehicle Technology (Vol. 5422, pp. 228–237).Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Pat Marion
    • 1
  • Maurice Fallon
    • 2
  • Robin Deits
    • 1
  • Andrés Valenzuela
    • 1
  • Claudia Pérez D’Arpino
    • 1
  • Greg Izatt
    • 1
  • Lucas Manuelli
    • 1
  • Matt Antone
    • 1
  • Hongkai Dai
    • 1
  • Twan Koolen
    • 1
  • John Carter
    • 1
  • Scott Kuindersma
    • 3
  • Russ Tedrake
    • 1
  1. 1.Computer Science and Artificial Intelligence LaboratoryMassachusetts Institute of TechnologyCambridgeUSA
  2. 2.Oxford Robotics InstituteUniversity of OxfordOxfordUK
  3. 3.John A. Paulson School of Engineering and Applied SciencesHarvard UniversityCambridgeUSA

Personalised recommendations