Skip to main content
SpringerLink
Log in

Online planning for human–multi-robot interactive theatrical performance

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

We propose and evaluate a multi-robot system designed to enable live, improvisational theatric performance through online interaction between a performer and a robot system. The proposed system translates theatric performer intent into dynamically feasible trajectories for multi-robot ensembles without requiring prior knowledge of the ordering or timing of the desired robot motions. We allow a user to issue detailed instructions composed of desired motion descriptors in an online setting to specify the motion of varying collectives of robots via a centralized system planner. The centralized planner refines user motion specifications into safe and dynamically feasible trajectories thereby reducing the cognitive burden placed on the performer. We evaluate the system on a team of aerial robots (quadrotors), and show through offline simulation and online performance that the proposed system formulation translates online input into non-colliding dynamically feasible trajectories enabling a fleet of fifteen quadrotors to perform a series of coordinated behaviors in response to improvised direction from a human operator.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. In practice, \(t_s\) is set to a value slightly ahead of the instruction receipt time to account for planning computation time, allowing robots to transition between trajectories without discontinuities.

  2. https://www.bitcraze.io/crazyflie-2/.

  3. http://www.andrew.cmu.edu/user/ecappo/AURO17.mp4.

References

  • Alonso-Mora, J., Lohaus, S.H., Leemann, P., Siegwart, R., & Beardsley, P. (2015). Gesture based human–multi-robot swarm interaction and its application to an interactive display. In Proceedings of the IEEE international conference on robotics and automation (pp. 5948–5953).

  • Alonso-Mora, J., Montijano, E., Schwager, M., & Rus, D. (2016). Distributed multi-robot formation control among obstacles: A geometric and optimization approach with consensus. In Proceedings of the IEEE international conference on robotics and automation (pp. 5356–5363).

  • Amato, C., Konidaris, G., Cruz, G., Maynor, C. A., How, J. P., & Kaelbling, L. P. (2015). Planning for decentralized control of multiple robots under uncertainty. In: Proceedings of the IEEE international conference on robotics and automation (pp. 1241–1248). IEEE.

  • Augugliaro, F., Schoellig, A. P., & D’Andrea, R.(2012). Generation of collision-free trajectories for a quadrocopter fleet: A sequential convex programming approach. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 1917–1922).

  • Augugliaro, F., Schoellig, A. P., & D’Andrea, R. (2013). Dance of the flying machines: Methods for designing and executing an aerial dance choreography. IEEE Robotics and Automation Magazine, 20(4), 96–104.

    Article  Google Scholar 

  • AURORA, YouTube Music, & Carnegie Mellon University.(2016). AURORA: Winter Bird https://www.youtube.com/watch?v=PYdm_Z1wnQ0. Accessed 31 January 2017.

  • Bashyal, S., & Venayagamoorthy, G. K. (2008). Human swarm interaction for radiation source search and localization. In IEEE swarm intelligence symposium (pp. 1–8). IEEE.

  • Brown, D. S., Kerman, S. C., & Goodrich, M. A. (2014). Human–swarm interactions based on managing attractors. In Proceedings of the ACM/IEEE international conference on human–robot interaction (pp. 90–97). Bielefeld, Germany .

  • Cappo, E. A., Desai, A., & Michael, N. (2016). Robust coordinated aerial deployments for theatrical applications given online user interaction via behavior composition: Distributed. Robotics and Autonomous Systems, 27, 171.

    Google Scholar 

  • Carpin, S., & Pagello, E.(2012). On parallel RRTs for multi-robot systems. In Proceedings of the 8th conference on Italian association for artificial intelligence (pp. 834–841).

  • Chamseddine, A., Zhang, Y., Rabbath, C. A., Join, C., & Theilliol, D. (2012). Flatness based trajectory planning/replanning for a quadrotor aerial vehicle. In IEEE transactions on aerospace and electronic systems (pp. 2832–2848).

  • Chen, Y., Cutler, M., & How, J. P. (2015). Decoupled multiagent path planning via incremental sequential convex programming. In Proceedings of the IEEE international conference on robotics and automation (pp. 5954–5961). IEEE, Seattle, WA.

  • Cirque du Soleil, ETH Zurich, and Verity Studios. Sparked: A live interaction between humans and quadcopters (2014). http://flyingmachinearena.org/sparked/. Accessed 14 July 2016.

  • Clarke, E. M., & Wing, J. M. (1996). Formal methods: State of the art and future directions. ACM Computing Surveys (CSUR), 28(4), 626–643.

    Article  Google Scholar 

  • Cormen, T. H., Stein, C., Rivest, R. L., & Leiserson, C. E. (2001). Introduction to algorithms (2nd ed.). New York: McGraw-Hill Higher Education.

    MATH  Google Scholar 

  • Coxworth, B. (2014). Cirque du Soleil and ETH Zurich collaborate on human/drone performance. Gizmag. http://www.gizmag.com/cirque-du-soleil-sparked-drone-video/33921/. Accessed 14 July 2016.

  • DeCastro, J. A., Alonso-Mora, J., Raman, V., Rus, D., & Kress-Gazit, H. (2018). Collision-free reactive mission and motion planning for multi-robot systems. In Robotics research (pp. 459–476). Springer.

  • Desai, A., Cappo, E. A., & Michael, N. (2016). Dynamically feasible and safe shape transitions for teams of aerial robots. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 5489–5494). Daejeon, Korea.

  • Doriya, R., Mishra, S., & Gupta, S. (2015). A brief survey and analysis of multi-robot communication and coordination. In 2015 International conference on computing, communication and automation (ICCCA) (pp. 1014–1021). IEEE .

  • Ehang. (2017). Chinese new year 2017 lantern festival. http://www.ehang.com/cn/news/228.html. Accessed 15 February 2017.

  • Ferguson, D., Kalra, N., & Stentz, A. (2006). Replanning with RRTs. In Proceedings of the IEEE international conference on robotics and automation (pp. 1243–1248). IEEE.

  • Gazi, V., Fidan, B., Marques, L., & Ordonez, R.(2015). Robot swarms: Dynamics and control. In Mobile robots for dynamic environments (pp. 81–85). ASME .

  • Giammarco, K., & Giles, K. (2018). Verification and validation of behavior models using lightweight formal methods. In Disciplinary convergence in systems engineering research (pp. 431–447). Springer.

  • Hauri, S., Alonso-Mora, J., Breitenmoser, A., Siegwart, R., & Beardsley, P. (2014). Multi-robot formation control via a real-time drawing interface. In Field and service robotics (pp. 175–189). Springer.

  • Häusler, A. J., Saccon, A., Aguiar, A. P., Hauser, J., & Pascoal, A. M. (2016). Energy-optimal motion planning for multiple robotic vehicles with collision avoidance. IEEE Transactions on Control Systems Technology, 24(3), 867–883.

    Article  Google Scholar 

  • Hehn, M., & D’Andrea, R.(2011). Quadcopter trajectory generation and control. In IFAC world congress (pp. 1485–1491).

  • Hoffman, G., Kubat, R., & Breazeal, C. (2008). A hybrid control system for puppeteering a live robotic stage actor. In RO-MAN 17th IEEE international symposium on robot and human interactive communication (pp. 354–359). IEEE.

  • Hoy, M., Matveev, A. S., & Savkin, A. V. (2015). Algorithms for collision-free navigation of mobile robots in complex cluttered environments: A survey. Robotica, 33(03), 463–497.

    Article  Google Scholar 

  • Huang, E. (2017). Watch a record-setting 1000 drones take to the sky in China to celebrate the lantern festival. https://qz.com/908927/watch-a-guinness-record-setting-1000-drone-show-in-guangzhou-china-celebrating-lantern-festival-and-end-of-chinese-new-year/. Accessed 15 February 2017.

  • Intel Corp., Lady Gaga, Pepsi, National Football League. (2017). Pepsi zero sugar super bowl li halftime show. http://www.intel.com/content/www/us/en/technology-innovation/aerial-technology-light-show.html. Accessed 15 February 2017.

  • Knight, H., & Gray, M. (2012). Acting lesson with robot: Emotional gestures. In Proceedings of the ACM/IEEE international conference on human–robot interaction (pp. 407–407). IEEE.

  • Kolling, A., Nunnally, S., & Lewis, M. (2012). Towards human control of robot swarms. In Proceedings of the ACM/IEEE international conference on human–robot interaction (pp. 89–96). ACM.

  • Kolling, A., Walker, P., Chakraborty, N., Sycara, K., & Lewis, M. (2016). Human interaction with robot swarms: A survey. IEEE Transactions on Human–Machine Systems, 46(1), 9–26.

    Article  Google Scholar 

  • Kress-Gazit, H., Lahijanian, M., & Raman, V. (2018). Synthesis for robots: Guarantees and feedback for robot behavior. Annual Review of Control, Robotics, and Autonomous Systems, 18, 23.

    Google Scholar 

  • Kuhn, H. (2012). The hungarian method for assignment problem. In Naval research logistics quaterly (pp. 83–97) .

  • Legay, A., Delahaye, B., & Bensalem, S.(2010). Statistical model checking: An overview. In International conference on runtime verification (pp. 122–135). Springer.

  • Leonard, N. E., Paley, D. A., Davis, R. E., Fratantoni, D. M., Lekien, F., & Zhang, F. (2010). Coordinated control of an underwater glider fleet in an adaptive ocean sampling field experiment in Monterey Bay. Journal of Field Robotics, 27(6), 718–740.

    Article  Google Scholar 

  • Levy, D., Sundar, K., & Rathinam, S. (2014). Heuristics for routing heterogeneous unmanned vehicles with fuel constraints. Mathematical Problems in Engineering. https://doi.org/10.1155/2014/131450

    Article  Google Scholar 

  • Lupashin, S., Hehn, M., Mueller, M. W., Schoellig, A. P., Sherback, M., & D’Andrea, R. (2014). A platform for aerial robotics research and demonstration: The flying machine arena. Mechatronics, 24(1), 41–54.

    Article  Google Scholar 

  • MagicLab, Rhizomatiks Research: DroneMagic–Behind the scenes. (2016a). https://www.youtube.com/watch?v=JEWXBEDAq60. Accessed 14 June 2016.

  • MagicLab, Rhizomatiks Research: MagicLab 24 drone flight. (2016b). http://www.magiclab.nyc/research/drone_magic/. Accessed 14 June 2016.

  • Mahony, R., Kumar, V., & Corke, P. (2012). Multirotor aerial vehicles: Modeling, estimation, and control of quadrotor. IEEE Robotics and Automation Magazine, 19(3), 20–32.

    Article  Google Scholar 

  • Mellinger, D., Kushleyev, A., & Kumar, V. (2012). Mixed-integer quadratic program trajectory generation for heterogeneous quadrotor teams. In Proceedings of the IEEE international conference on robotics and automation (pp. 477–483). Saint Paul, MN.

  • Mitchell, D., Cappo, E. A., & Michael, N. (2016). Persistent robot formation flight via online substitution. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 4810–4815). IEEE, Daejeon, Korea.

  • Pourmehr, S., Monajjemi, V. M., & Vaughan, R., Mori, G. (2013). “You two! Take off!”: Creating, modifying and commanding groups of robots using face engagement and indirect speech in voice commands. In Proceedings of the IEEE international conference on robotics and automation (pp. 137–142). IEEE.

  • Preiss, J. A., Hönig, W., Sukhatme, G. S., & Ayanian, N. (2016). Crazyswarm: A large nano-quadcopter swarm

  • Ratner, D., & Warmuth, M. K. (1986). Finding a shortest solution for the N\(\times \)N extension of the 15-PUZZLE is intractable. In AAAI (pp. 168–172).

  • Riazi, S., Bengtsson, K., Wigström, O., Vidarsson, E., & Lennartson, B. (2015). Energy optimization of multi-robot systems. In 2015 IEEE International conference on automation science and engineering (CASE) (pp. 1345–1350). IEEE.

  • Richter, C., Bry, A., & Roy, N.(2013). Polynomial trajectory planning for aggressive quadrotor flight in dense indoor environments. In Proceedings of the IEEE international conference on robotics and automation.

  • Richter, C., Bry, A., & Roy, N. (2013). Polynomial trajectory planning for quadrotor flight. In Proceedings of the Robotics: Science and System.

  • Röger, G., & Helmert, M. (2012). Non-optimal multi-agent pathfinding is solved (since 1984). In SOCS.

  • Saha, I., Ramaithitima, R., Kumar, V., Pappas, G. J., & Seshia, S. A. (2014). Automated composition of motion primitives for multi-robot systems from safe LTL specifications. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 1525–1532). IEEE.

  • Saha, I., Ramaithitima, R., Kumar, V., Pappas, G. J., & Seshia, S. A. (2016). Implan: Scalable incremental motion planning for multi-robot systems. In ACM/IEEE 7th International conference on cyber-physical system (pp. 1–10). IEEE.

  • Secchi, C., Sabattini, L., & Fantuzzi, C.(2015). Conducting multi-robot systems: Gestures for the passive teleoperation of multiple slaves. InProceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 2803–2808). IEEE.

  • Setter, T., Fouraker, A., Kawashima, H., & Egerstedt, M. (2015). Haptic interactions with multi-robot swarms using manipulability. Journal of Human–Robot Interaction, 4(1), 60–74.

    Article  Google Scholar 

  • Sharon, G., Stern, R., Felner, A., & Sturtevant, N. R. (2012). Meta-agent conflict-based search for optimal multi-agent path finding. SOCS, 1, 39–40.

    MATH  Google Scholar 

  • Standley, T. S. (2010). Finding optimal solutions to cooperative pathfinding problems. AAAI, 1, 28–29.

    Google Scholar 

  • Stoica, A., Theodoridis, T., Hu, H., McDonald-Maier, K., & Barrero, D. F. (2013). Towards human-friendly efficient control of multi-robot teams. In International conference on collaboration technologies and systems (pp. 226–231). IEEE.

  • The Creators Project: Behind the scenes of a flying robot-built cocoon. (2016). https://youtu.be/xcVoS6DpFf8. Accessed 31 January 2017.

  • Turpin, M., Michael, N., & Kumar, V. (2013). Concurrent assignment and planning of trajectories for large teams of interchangeable robots. In Proceedings of the IEEE international conference on robotics and automation

  • Turpin, M., Mohta, K., Michael, N., & Kumar, V. (2013). Goal assignment and trajectory planning for large teams of aerial robots. In Proceedings of the robotics: science and system

  • Wagner, G., & Choset, H. (2015). Subdimensional expansion for multirobot path planning. Artificial Intelligence, 219, 1–24.

    Article  MathSciNet  Google Scholar 

  • Zhou, D., & Schwager, M. (2015). Virtual rigid bodies for coordinated agile maneuvering of teams of micro aerial vehicles. In Proceedings of the IEEE international conference on robotics and automation (pp. 1737–1742). Seattle, WA.

  • Zhou, D., & Schwager, M. (2016). Assistive collision avoidance for quadrotor swarm teleoperation. In Proceedings of the IEEE international conference on robotics and automation (pp. 1249–1254). Stockholm, Sweden.

Download references

Author information

Authors and Affiliations

  1. The Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA

    Ellen A. Cappo, Arjav Desai, Matthew Collins & Nathan Michael

Authors
  1. Ellen A. Cappo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arjav Desai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matthew Collins
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nathan Michael
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ellen A. Cappo.

Additional information

This is one of several papers published in Autonomous Robots comprising the “Special Issue on Distributed Robotics: From Fundamentals to Applications”.

We gratefully acknowledge support from ONR Grants N00014-13-1-0821 and N00014-15-1-2929.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 160236 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cappo, E.A., Desai, A., Collins, M. et al. Online planning for human–multi-robot interactive theatrical performance. Auton Robot 42, 1771–1786 (2018). https://doi.org/10.1007/s10514-018-9755-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-018-9755-0

Keywords

Search

Navigation