Advertisement

Autonomous Robots

, Volume 42, Issue 8, pp 1771–1786 | Cite as

Online planning for human–multi-robot interactive theatrical performance

  • Ellen A. CappoEmail author
  • Arjav Desai
  • Matthew Collins
  • Nathan Michael
Article
  • 386 Downloads
Part of the following topical collections:
  1. Special Issue on Distributed Robotics: From Fundamentals to Applications

Abstract

We propose and evaluate a multi-robot system designed to enable live, improvisational theatric performance through online interaction between a performer and a robot system. The proposed system translates theatric performer intent into dynamically feasible trajectories for multi-robot ensembles without requiring prior knowledge of the ordering or timing of the desired robot motions. We allow a user to issue detailed instructions composed of desired motion descriptors in an online setting to specify the motion of varying collectives of robots via a centralized system planner. The centralized planner refines user motion specifications into safe and dynamically feasible trajectories thereby reducing the cognitive burden placed on the performer. We evaluate the system on a team of aerial robots (quadrotors), and show through offline simulation and online performance that the proposed system formulation translates online input into non-colliding dynamically feasible trajectories enabling a fleet of fifteen quadrotors to perform a series of coordinated behaviors in response to improvised direction from a human operator.

Keywords

Human–multi-robot interaction Multi-robot online planning Multi-robot formation trajectory generation 

Supplementary material

Supplementary material 1 (mp4 160236 KB)

References

  1. Alonso-Mora, J., Lohaus, S.H., Leemann, P., Siegwart, R., & Beardsley, P. (2015). Gesture based human–multi-robot swarm interaction and its application to an interactive display. In Proceedings of the IEEE international conference on robotics and automation (pp. 5948–5953).Google Scholar
  2. Alonso-Mora, J., Montijano, E., Schwager, M., & Rus, D. (2016). Distributed multi-robot formation control among obstacles: A geometric and optimization approach with consensus. In Proceedings of the IEEE international conference on robotics and automation (pp. 5356–5363).Google Scholar
  3. Amato, C., Konidaris, G., Cruz, G., Maynor, C. A., How, J. P., & Kaelbling, L. P. (2015). Planning for decentralized control of multiple robots under uncertainty. In: Proceedings of the IEEE international conference on robotics and automation (pp. 1241–1248). IEEE.Google Scholar
  4. Augugliaro, F., Schoellig, A. P., & D’Andrea, R.(2012). Generation of collision-free trajectories for a quadrocopter fleet: A sequential convex programming approach. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 1917–1922).Google Scholar
  5. Augugliaro, F., Schoellig, A. P., & D’Andrea, R. (2013). Dance of the flying machines: Methods for designing and executing an aerial dance choreography. IEEE Robotics and Automation Magazine, 20(4), 96–104.CrossRefGoogle Scholar
  6. AURORA, YouTube Music, & Carnegie Mellon University.(2016). AURORA: Winter Bird https://www.youtube.com/watch?v=PYdm_Z1wnQ0. Accessed 31 January 2017.
  7. Bashyal, S., & Venayagamoorthy, G. K. (2008). Human swarm interaction for radiation source search and localization. In IEEE swarm intelligence symposium (pp. 1–8). IEEE.Google Scholar
  8. Brown, D. S., Kerman, S. C., & Goodrich, M. A. (2014). Human–swarm interactions based on managing attractors. In Proceedings of the ACM/IEEE international conference on human–robot interaction (pp. 90–97). Bielefeld, Germany .Google Scholar
  9. Cappo, E. A., Desai, A., & Michael, N. (2016). Robust coordinated aerial deployments for theatrical applications given online user interaction via behavior composition: Distributed. Robotics and Autonomous Systems, 27, 171.Google Scholar
  10. Carpin, S., & Pagello, E.(2012). On parallel RRTs for multi-robot systems. In Proceedings of the 8th conference on Italian association for artificial intelligence (pp. 834–841).Google Scholar
  11. Chamseddine, A., Zhang, Y., Rabbath, C. A., Join, C., & Theilliol, D. (2012). Flatness based trajectory planning/replanning for a quadrotor aerial vehicle. In IEEE transactions on aerospace and electronic systems (pp. 2832–2848).Google Scholar
  12. Chen, Y., Cutler, M., & How, J. P. (2015). Decoupled multiagent path planning via incremental sequential convex programming. In Proceedings of the IEEE international conference on robotics and automation (pp. 5954–5961). IEEE, Seattle, WA.Google Scholar
  13. Cirque du Soleil, ETH Zurich, and Verity Studios. Sparked: A live interaction between humans and quadcopters (2014). http://flyingmachinearena.org/sparked/. Accessed 14 July 2016.
  14. Clarke, E. M., & Wing, J. M. (1996). Formal methods: State of the art and future directions. ACM Computing Surveys (CSUR), 28(4), 626–643.CrossRefGoogle Scholar
  15. Cormen, T. H., Stein, C., Rivest, R. L., & Leiserson, C. E. (2001). Introduction to algorithms (2nd ed.). New York: McGraw-Hill Higher Education.zbMATHGoogle Scholar
  16. Coxworth, B. (2014). Cirque du Soleil and ETH Zurich collaborate on human/drone performance. Gizmag. http://www.gizmag.com/cirque-du-soleil-sparked-drone-video/33921/. Accessed 14 July 2016.
  17. DeCastro, J. A., Alonso-Mora, J., Raman, V., Rus, D., & Kress-Gazit, H. (2018). Collision-free reactive mission and motion planning for multi-robot systems. In Robotics research (pp. 459–476). Springer.Google Scholar
  18. Desai, A., Cappo, E. A., & Michael, N. (2016). Dynamically feasible and safe shape transitions for teams of aerial robots. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 5489–5494). Daejeon, Korea.Google Scholar
  19. Doriya, R., Mishra, S., & Gupta, S. (2015). A brief survey and analysis of multi-robot communication and coordination. In 2015 International conference on computing, communication and automation (ICCCA) (pp. 1014–1021). IEEE .Google Scholar
  20. Ehang. (2017). Chinese new year 2017 lantern festival. http://www.ehang.com/cn/news/228.html. Accessed 15 February 2017.
  21. Ferguson, D., Kalra, N., & Stentz, A. (2006). Replanning with RRTs. In Proceedings of the IEEE international conference on robotics and automation (pp. 1243–1248). IEEE.Google Scholar
  22. Gazi, V., Fidan, B., Marques, L., & Ordonez, R.(2015). Robot swarms: Dynamics and control. In Mobile robots for dynamic environments (pp. 81–85). ASME .Google Scholar
  23. Giammarco, K., & Giles, K. (2018). Verification and validation of behavior models using lightweight formal methods. In Disciplinary convergence in systems engineering research (pp. 431–447). Springer.Google Scholar
  24. Hauri, S., Alonso-Mora, J., Breitenmoser, A., Siegwart, R., & Beardsley, P. (2014). Multi-robot formation control via a real-time drawing interface. In Field and service robotics (pp. 175–189). Springer.Google Scholar
  25. Häusler, A. J., Saccon, A., Aguiar, A. P., Hauser, J., & Pascoal, A. M. (2016). Energy-optimal motion planning for multiple robotic vehicles with collision avoidance. IEEE Transactions on Control Systems Technology, 24(3), 867–883.CrossRefGoogle Scholar
  26. Hehn, M., & D’Andrea, R.(2011). Quadcopter trajectory generation and control. In IFAC world congress (pp. 1485–1491).Google Scholar
  27. Hoffman, G., Kubat, R., & Breazeal, C. (2008). A hybrid control system for puppeteering a live robotic stage actor. In RO-MAN 17th IEEE international symposium on robot and human interactive communication (pp. 354–359). IEEE.Google Scholar
  28. Hoy, M., Matveev, A. S., & Savkin, A. V. (2015). Algorithms for collision-free navigation of mobile robots in complex cluttered environments: A survey. Robotica, 33(03), 463–497.CrossRefGoogle Scholar
  29. Huang, E. (2017). Watch a record-setting 1000 drones take to the sky in China to celebrate the lantern festival. https://qz.com/908927/watch-a-guinness-record-setting-1000-drone-show-in-guangzhou-china-celebrating-lantern-festival-and-end-of-chinese-new-year/. Accessed 15 February 2017.
  30. Intel Corp., Lady Gaga, Pepsi, National Football League. (2017). Pepsi zero sugar super bowl li halftime show. http://www.intel.com/content/www/us/en/technology-innovation/aerial-technology-light-show.html. Accessed 15 February 2017.
  31. Knight, H., & Gray, M. (2012). Acting lesson with robot: Emotional gestures. In Proceedings of the ACM/IEEE international conference on human–robot interaction (pp. 407–407). IEEE.Google Scholar
  32. Kolling, A., Nunnally, S., & Lewis, M. (2012). Towards human control of robot swarms. In Proceedings of the ACM/IEEE international conference on human–robot interaction (pp. 89–96). ACM.Google Scholar
  33. Kolling, A., Walker, P., Chakraborty, N., Sycara, K., & Lewis, M. (2016). Human interaction with robot swarms: A survey. IEEE Transactions on Human–Machine Systems, 46(1), 9–26.CrossRefGoogle Scholar
  34. Kress-Gazit, H., Lahijanian, M., & Raman, V. (2018). Synthesis for robots: Guarantees and feedback for robot behavior. Annual Review of Control, Robotics, and Autonomous Systems, 18, 23.Google Scholar
  35. Kuhn, H. (2012). The hungarian method for assignment problem. In Naval research logistics quaterly (pp. 83–97) .Google Scholar
  36. Legay, A., Delahaye, B., & Bensalem, S.(2010). Statistical model checking: An overview. In International conference on runtime verification (pp. 122–135). Springer.Google Scholar
  37. Leonard, N. E., Paley, D. A., Davis, R. E., Fratantoni, D. M., Lekien, F., & Zhang, F. (2010). Coordinated control of an underwater glider fleet in an adaptive ocean sampling field experiment in Monterey Bay. Journal of Field Robotics, 27(6), 718–740.CrossRefGoogle Scholar
  38. Levy, D., Sundar, K., & Rathinam, S. (2014). Heuristics for routing heterogeneous unmanned vehicles with fuel constraints. Mathematical Problems in Engineering.  https://doi.org/10.1155/2014/131450 CrossRefGoogle Scholar
  39. Lupashin, S., Hehn, M., Mueller, M. W., Schoellig, A. P., Sherback, M., & D’Andrea, R. (2014). A platform for aerial robotics research and demonstration: The flying machine arena. Mechatronics, 24(1), 41–54.CrossRefGoogle Scholar
  40. MagicLab, Rhizomatiks Research: DroneMagic–Behind the scenes. (2016a). https://www.youtube.com/watch?v=JEWXBEDAq60. Accessed 14 June 2016.
  41. MagicLab, Rhizomatiks Research: MagicLab 24 drone flight. (2016b). http://www.magiclab.nyc/research/drone_magic/. Accessed 14 June 2016.
  42. Mahony, R., Kumar, V., & Corke, P. (2012). Multirotor aerial vehicles: Modeling, estimation, and control of quadrotor. IEEE Robotics and Automation Magazine, 19(3), 20–32.CrossRefGoogle Scholar
  43. Mellinger, D., Kushleyev, A., & Kumar, V. (2012). Mixed-integer quadratic program trajectory generation for heterogeneous quadrotor teams. In Proceedings of the IEEE international conference on robotics and automation (pp. 477–483). Saint Paul, MN.Google Scholar
  44. Mitchell, D., Cappo, E. A., & Michael, N. (2016). Persistent robot formation flight via online substitution. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 4810–4815). IEEE, Daejeon, Korea.Google Scholar
  45. Pourmehr, S., Monajjemi, V. M., & Vaughan, R., Mori, G. (2013). “You two! Take off!”: Creating, modifying and commanding groups of robots using face engagement and indirect speech in voice commands. In Proceedings of the IEEE international conference on robotics and automation (pp. 137–142). IEEE.Google Scholar
  46. Preiss, J. A., Hönig, W., Sukhatme, G. S., & Ayanian, N. (2016). Crazyswarm: A large nano-quadcopter swarmGoogle Scholar
  47. Ratner, D., & Warmuth, M. K. (1986). Finding a shortest solution for the N\(\times \)N extension of the 15-PUZZLE is intractable. In AAAI (pp. 168–172).Google Scholar
  48. Riazi, S., Bengtsson, K., Wigström, O., Vidarsson, E., & Lennartson, B. (2015). Energy optimization of multi-robot systems. In 2015 IEEE International conference on automation science and engineering (CASE) (pp. 1345–1350). IEEE.Google Scholar
  49. Richter, C., Bry, A., & Roy, N.(2013). Polynomial trajectory planning for aggressive quadrotor flight in dense indoor environments. In Proceedings of the IEEE international conference on robotics and automation.Google Scholar
  50. Richter, C., Bry, A., & Roy, N. (2013). Polynomial trajectory planning for quadrotor flight. In Proceedings of the Robotics: Science and System.Google Scholar
  51. Röger, G., & Helmert, M. (2012). Non-optimal multi-agent pathfinding is solved (since 1984). In SOCS.Google Scholar
  52. Saha, I., Ramaithitima, R., Kumar, V., Pappas, G. J., & Seshia, S. A. (2014). Automated composition of motion primitives for multi-robot systems from safe LTL specifications. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 1525–1532). IEEE.Google Scholar
  53. Saha, I., Ramaithitima, R., Kumar, V., Pappas, G. J., & Seshia, S. A. (2016). Implan: Scalable incremental motion planning for multi-robot systems. In ACM/IEEE 7th International conference on cyber-physical system (pp. 1–10). IEEE.Google Scholar
  54. Secchi, C., Sabattini, L., & Fantuzzi, C.(2015). Conducting multi-robot systems: Gestures for the passive teleoperation of multiple slaves. InProceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 2803–2808). IEEE.Google Scholar
  55. Setter, T., Fouraker, A., Kawashima, H., & Egerstedt, M. (2015). Haptic interactions with multi-robot swarms using manipulability. Journal of Human–Robot Interaction, 4(1), 60–74.CrossRefGoogle Scholar
  56. Sharon, G., Stern, R., Felner, A., & Sturtevant, N. R. (2012). Meta-agent conflict-based search for optimal multi-agent path finding. SOCS, 1, 39–40.zbMATHGoogle Scholar
  57. Standley, T. S. (2010). Finding optimal solutions to cooperative pathfinding problems. AAAI, 1, 28–29.Google Scholar
  58. Stoica, A., Theodoridis, T., Hu, H., McDonald-Maier, K., & Barrero, D. F. (2013). Towards human-friendly efficient control of multi-robot teams. In International conference on collaboration technologies and systems (pp. 226–231). IEEE.Google Scholar
  59. The Creators Project: Behind the scenes of a flying robot-built cocoon. (2016). https://youtu.be/xcVoS6DpFf8. Accessed 31 January 2017.
  60. Turpin, M., Michael, N., & Kumar, V. (2013). Concurrent assignment and planning of trajectories for large teams of interchangeable robots. In Proceedings of the IEEE international conference on robotics and automation Google Scholar
  61. Turpin, M., Mohta, K., Michael, N., & Kumar, V. (2013). Goal assignment and trajectory planning for large teams of aerial robots. In Proceedings of the robotics: science and system Google Scholar
  62. Wagner, G., & Choset, H. (2015). Subdimensional expansion for multirobot path planning. Artificial Intelligence, 219, 1–24.MathSciNetCrossRefGoogle Scholar
  63. Zhou, D., & Schwager, M. (2015). Virtual rigid bodies for coordinated agile maneuvering of teams of micro aerial vehicles. In Proceedings of the IEEE international conference on robotics and automation (pp. 1737–1742). Seattle, WA.Google Scholar
  64. Zhou, D., & Schwager, M. (2016). Assistive collision avoidance for quadrotor swarm teleoperation. In Proceedings of the IEEE international conference on robotics and automation (pp. 1249–1254). Stockholm, Sweden.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Ellen A. Cappo
    • 1
    Email author
  • Arjav Desai
    • 1
  • Matthew Collins
    • 1
  • Nathan Michael
    • 1
  1. 1.The Robotics InstituteCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations