Skip to main content
SpringerLink
Log in

On multi-human multi-robot remote interaction: a study of transparency, inter-human communication, and information loss in remote interaction

  • Published:
Swarm Intelligence Aims and scope Submit manuscript

Abstract

In this paper, we investigate how to design an effective interface for remote multi-human–multi-robot interaction. While significant research exists on interfaces for individual human operators, little research exists for the multi-human case. Yet, this is a critical problem to solve to make complex, large-scale missions achievable in which direct operator involvement is impossible or undesirable, and robot swarms act as a semi-autonomous agents. This paper’s contribution is twofold. The first contribution is an exploration of the design space of computer-based interfaces for multi-human multi-robot operations. In particular, we focus on agent transparency and on the factors that affect inter-human communication in ideal conditions, i.e., without communication issues. Our second contribution concerns the same problem, but considering increasing degrees of information loss, defined as intermittent reception of data with noticeable gaps between individual receipts. We derived a set of design recommendations based on two user studies involving 48 participants.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24

Similar content being viewed by others

Notes

  1. https://aws.amazon.com/.

  2. https://nodejs.org/.

  3. https://get.webgl.org/.

  4. https://github.com/NESTLab/argos3-webviz.

  5. https://www.zoom.us.

References

  • Admoni, H., & Scassellati, B. (2017). Social eye gaze in human–robot interaction: A review. Journal of Human-Robot Interaction, 6(1), 25–63.

    Article  Google Scholar 

  • Ayanian, N., Spielberg, A., Arbesfeld, M., Strauss, J., & Rus, D. (2014). Controlling a team of robots with a single input. In 2014 IEEE International Conference on Robotics and Automation (ICRA) (pp. 1755–1762). IEEE. http://ieeexplore.ieee.org/abstract/document/6907088/

  • Bhaskara, A., Skinner, M., & Loft, S. (2020). Agent transparency: A review of current theory and evidence. IEEE Transactions on Human-Machine Systems. https://doi.org/10.1109/THMS.2020.2965529.

    Article  Google Scholar 

  • Black, D. (1976). Partial justification of the borda count. Public Choice, 28, 1–15.

    Article  Google Scholar 

  • Breazeal, C., Kidd, C. D., Thomaz, A. L., Hoffman, G., & Berlin, M. (2005). Effects of nonverbal communication on efficiency and robustness in human–robot teamwork. In 2005 IEEE/RSJ international conference on intelligent robots and systems (pp. 708–713). IEEE.

  • Capelli, B., Secchi, C., & Sabattini, L. (2019). Communication through motion: Legibility of multi-robot systems. In 2019 international symposium on Multi-Robot and Multi-Agent Systems (MRS) (pp. 126–132). IEEE. https://doi.org/10.1109/MRS.2019.8901100

  • Chakraborti, T., Kulkarni, A., Sreedharan, S., Smith, D. E., & Kambhampati, S. (2019). Explicability? Legibility? Predictability? Transparency? Privacy? Security? The emerging landscape of interpretable agent behavior. In Proceedings of the international conference on automated planning and scheduling, Vol. 29, pp. 86–96.

  • Che, Y., Okamura, A. M., & Sadigh, D. (2020). Efficient and trustworthy social navigation via explicit and implicit robot–human communication. IEEE Transactions on Robotics, 36, 692–707.

    Article  Google Scholar 

  • Chen, J.Y., Procci, K., Boyce, M., Wright, J., Garcia, A., & Barnes, M. (2014). Situation awareness-based agent transparency. Tech. rep., Defense Technical Information Center, Fort Belvoir, VA. https://doi.org/10.21236/ADA600351

  • Chen, J. Y. C., Haas, E. C., & Barnes, M. J. (2007). Human performance issues and user interface design for teleoperated robots. IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 37(6), 1231–1245. https://doi.org/10.1109/TSMCC.2007.905819.

    Article  Google Scholar 

  • Chen, J. Y. C., Lakhmani, S. G., Stowers, K., Selkowitz, A. R., Wright, J. L., & Barnes, M. (2018). Situation awareness-based agent transparency and human-autonomy teaming effectiveness. Theoretical Issues in Ergonomics Science, 19(3), 259–282.

    Article  Google Scholar 

  • Cheung, Y., & Chung, J. H. (2011). Semi-autonomous control of single-master multi-slave teleoperation of heterogeneous robots for multi-task multi-target pairing. International Journal of Control and Automation, 4(3), 17.

    Google Scholar 

  • Daily, M., Cho, Y., Martin, K., & Payton, D. (2003). World embedded interfaces for human–robot interaction. In Proceedings of the 36th annual Hawaii International Conference on System Sciences, 2003 (p. 6). IEEE.

  • Dardona, T., Eslamian, S., Reisner, L. A., & Pandya, A. (2019). Remote presence: Development and usability evaluation of a head-mounted display for camera control on the da Vinci surgical system. Robotics, 8(2), 31.

    Article  Google Scholar 

  • Dimitoglou, G. (2019). Telepresence: Evaluation of robot stand-ins for remote student learning. Journal of Computing Sciences in Colleges, 35, 15.

    Google Scholar 

  • Do, N. D., Yamashina, Y., & Namerikawa, T. (2011). Multiple cooperative bilateral teleoperation with time-varying delay. SICE Journal of Control, Measurement, and System Integration, 4(2), 89–96.

    Article  Google Scholar 

  • Dong, G. L., Gun, R. C., Min, S. L., Kim, B.-S., Oh, S., & Son H. I. (2013). Human-centered evaluation of multi-user teleoperation for mobile manipulator in unmanned offshore plants. In 2013 IEEE/RSJ international conference on intelligent robots and systems (pp. 5431–5438). IEEE. https://doi.org/10.1109/IROS.2013.6697142

  • Ellis, S. R., Mania, K., Adelstein, B. D., & Hill, M. I. (2004). Generalizeability of latency detection in a variety of virtual environments. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 48, pp. 2632–2636). SAGE Publications.

  • Endsley, M. R., & Kiris, E. O. (1995). The out-of-the-loop performance problem and level of control in automation. Human Factors, 37(2), 381–394.

    Article  Google Scholar 

  • Esfahlani, S. S. (2019). Mixed reality and remote sensing application of unmanned aerial vehicle in fire and smoke detection. Journal of Industrial Information Integration. https://doi.org/10.1016/j.jii.2019.04.006.

    Article  Google Scholar 

  • Ferreira, L. R. N., & Pereira, L. T. (2020). Immersive mobile telepresence systems: A systematic literature review. Journal of Mobile Multimedia, 15, 16.

    Google Scholar 

  • Feth, D., Tran, B. A., Groten, R., Peer, A., & Buss, M. (2009). Shared-control paradigms in multi-operator-single-robot teleoperation. In R. Dillmann, D. Vernon, Y. Nakamura, S. Schaal, H. Ritter, G. Sagerer, R. Dillmann, M. Buss (Eds.), Human centered robot systems. Series Title: Cognitive Systems Monographs (Vol. 6, pp. 53–62). Springer. https://doi.org/10.1007/978-3-642-10403-9_6

  • Friedman, M. (1937). The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32(200), 675–701.

    Article  Google Scholar 

  • Ghiringhelli, F., Guzzi, J., Di Caro, G. A., Caglioti, V., Gambardella, L. M., & Giusti, A. (2014). Interactive augmented reality for understanding and analyzing multi-robot systems. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2014) (pp. 1195–1201). IEEE.

  • Goertz, R. C., & Thompson, W. M. (1954). Electronically controlled manipulator (Vol. 12). Nucleonics (US) Ceased Publication.

  • Gouraud, J., Delorme, A., & Berberian, B. (2017). Autopilot, mind wandering, and the out of the loop performance problem. Frontiers in Neuroscience, 11, 541.

    Article  Google Scholar 

  • Hart, S. G., & Staveland, L. E. (1988). Development of nasa-tlx (task load index): Results of empirical and theoretical research. In Advances in psychology (Vol. 52, pp. 139–183). Elsevier.

  • Hokayem, P. F., & Spong, M. W. (2006). Bilateral teleoperation: An historical survey. Automatica, 42(12), 2035–2057.

    Article  MathSciNet  Google Scholar 

  • Holdcroft, D. (1976). Forms of indirect communication: An outline. Philosophy& Rhetoric, 9, 147–161.

    Google Scholar 

  • Hong, A., Bulthoff, H. H., & Son, H. I. (2013). A visual and force feedback for multi-robot teleoperation in outdoor environments: A preliminary result. In 2013 IEEE International Conference on Robotics and Automation (pp. 1471–1478). IEEE. https://doi.org/10.1109/ICRA.2013.6630765

  • Jingtai, L., Sun, L., Chen, T., Huang, X., & Zhao, C. (2005). Competitive multi-robot teleoperation. In Proceedings of the 2005 IEEE International Conference on Robotics and Automation (pp. 75–80). IEEE. https://doi.org/10.1109/ROBOT.2005.1570099

  • Jung, H., & Song, Y. E. (2018). Robotic remote control based on human motion via virtual collaboration system: A survey. Journal of Advanced Mechanical Design, Systems, and Manufacturing, 12(7), JAMDSM0126. https://doi.org/10.1299/jamdsm.2018jamdsm0126.

    Article  Google Scholar 

  • Kim, J., Oh, S., & Kim, D. (2012). Implementation of a work distribution function for tele-operation under multi-user and multi-robot environments. In 2012 12th international conference on control, automation and systems p. 4.

  • Kim, J., Yun, S., Yoo, J., & Kim, D. (2013). Implementation of N:1/1:N work distribution function for tele-operation under multi-user and multi-robot environments. In 2013 10th international conference on Ubiquitous Robots and Ambient Intelligence (URAI) (pp. 328–330). IEEE. https://doi.org/10.1109/URAI.2013.6677378

  • Kolling, A., Sycara, K., Nunnally, S., & Lewis, M. (2013). Human swarm interaction: An experimental study of two types of interaction with foraging swarms. Journal of Human-Robot Interaction. https://doi.org/10.5898/JHRI.2.2.Kolling.

    Article  Google Scholar 

  • Lakhmani, S. G., Wright, J. L., Schwartz, M. R., & Barber, D. (2019). Exploring the effect of communication patterns and transparency on performance in a human–robot team. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 63(1), 160–164.

    Article  Google Scholar 

  • Lane, J. C., Carignan, C. R., Sullivan, B. R., Akin, D. L., Hunt, T., & Cohen, R. (2002). Effects of time delay on telerobotic control of neutral buoyancy vehicles. In Proceedings 2002 IEEE international conference on robotics and automation (Cat. No. 02CH37292) (Vol. 3, pp. 2874–2879). IEEE.

  • Lee, D., Franchi, A., Son, H. I., Ha, C., Bulthoff, H. H., & Giordano, P. R. (2013). Semiautonomous haptic teleoperation control architecture of multiple unmanned aerial vehicles. IEEE/ASME Transactions on Mechatronics, 18(4), 1334–1345.

    Article  Google Scholar 

  • Li, H., Zhang, L., & Kawashima, K. (2018). Operator dynamics for stability condition in haptic and teleoperation system: A survey. The International Journal of Medical Robotics and Computer Assisted Surgery, 14(2), e1881.

    Article  Google Scholar 

  • Lichiardopol, S. (2007). A survey on teleoperation. Technische Universitat Eindhoven, DCT report, Vol. 20, p. 34.

  • Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 140, 55.

    Google Scholar 

  • Liu, Z., Wu, M., Cao, W., Chen, L., Xu, J., Zhang, R., et al. (2017). A facial expression emotion recognition based human–robot interaction system. IEEE/CAA Journal of Automatica Sinica, 4, 668–676.

    Article  Google Scholar 

  • Lunghi, G., Marin, R., Di Castro, M., Masi, A., & Sanz, P. J. (2019). Multimodal human–robot interface for accessible remote robotic interventions in hazardous environments. IEEE Access, 7, 127290–127319.

    Article  Google Scholar 

  • Lyons, J. B. (2013). Being transparent about transparency: A model for human–robot interaction. In 2013 AAAI Spring Symposium Series.

  • Ma, L., Yan, J., Zhao, J., Chen, Z., & Cai, H. (2010). Teleoperation system of Internet-based multi-operator multi-mobile-manipulator. In 2010 International Conference on Electrical and Control Engineering (pp. 2236–2240). IEEE. https://doi.org/10.1109/iCECE.2010.551

  • MacKenzie, I. S., & Ware, C. (1993). Lag as a determinant of human performance in interactive systems. In Proceedings of the INTERACT’93 and CHI’93 conference on Human factors in computing systems, pp. 488–493.

  • Mavridis, N. (2015). A review of verbal and non-verbal human–robot interactive communication. Robotics and Autonomous Systems, 63, 22–35.

    Article  MathSciNet  Google Scholar 

  • Mercado, J. E., Rupp, M. A., Chen, J. Y. C., Barnes, M. J., Barber, D., & Procci, K. (2016). Intelligent agent transparency in human-agent teaming for multi-UxV management. Human Factors: The Journal of the Human Factors and Ergonomics Society, 58(3), 401–415.

    Article  Google Scholar 

  • Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81.

    Article  Google Scholar 

  • Murphy, R. R. (2012). A decade of rescue robots. In 2012 IEEE/RSJ international conference on intelligent robots and systems (pp. 5448–5449). IEEE.

  • Music, S., Salvietti, G., Dohmann, P. B., Chinello, F., Prattichizzo, D., & Hirche, S. (2019). Human–robot team interaction through wearable haptics for cooperative manipulation. IEEE Transactions on Haptics, 12(3), 350–362.

    Article  Google Scholar 

  • Nak Y. C., Kotoku, T., Ohba, K., Komoriya, K., Matsuhira, N., & Tanie, K. (2000). Remote coordinated controls in multiple telerobot cooperation. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065) (Vol. 4, pp. 3138–3143). IEEE. https://doi.org/10.1109/ROBOT.2000.845146

  • Patel, J., Ramaswamy, T., Li, Z., & Pinciroli, C. (2021). Direct and indirect communication in multi-human multi-robot interaction. IEEE Transactions on Human-Machine Systems (Submitted).

  • Patel, J., Xu, Y., & Pinciroli, C. (2019). Mixed-granularity human-swarm interaction. In 2019 IEEE International Conference on Robotics and Automation (ICRA). IEEE. http://arxiv.org/abs/1901.08522

  • Patel, T. M., Shah, S. C., & Pancholy, S. B. (2019). Long distance tele-robotic-assisted percutaneous coronary intervention: A report of first-in-human experience. EClinicalMedicine, 14, 53–58.

    Article  Google Scholar 

  • Pinciroli, C., Trianni, V., O’Grady, R., Pini, G., Brutschy, A., Brambilla, M., et al. (2012). ARGoS: A modular, parallel, multi-engine simulator for multi-robot systems. Swarm Intelligence, 6(4), 271–295.

    Article  Google Scholar 

  • Rakita, D., Mutlu, B., & Gleicher, M. (2019). Remote telemanipulation with adapting viewpoints in visually complex environments. In Robotics: Science and Systems XV. Robotics: Science and Systems Foundation. https://doi.org/10.15607/RSS.2019.XV.068

  • Ricks, B., Nielsen, C. W., & Goodrich, M. A. (2004). Ecological displays for robot interaction: A new perspective. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No. 04CH37566) (Vol. 3, pp. 2855–2860). IEEE.

  • Roldan, J. J., Pena-Tapia, E., Garcia-Aunon, P., Del Cerro, J., & Barrientos, A. (2019). Bringing adaptive and immersive interfaces to real-world multi-robot scenarios: Application to surveillance and intervention in infrastructures. IEEE Access, 7, 86319–86335.

    Article  Google Scholar 

  • Roundtree, K. A., Goodrich, M. A., & Adams, J. A. (2019). Transparency: Transitioning from human-machine systems to human-swarm systems. Journal of Cognitive Engineering and Decision Making. https://doi.org/10.1177/1555343419842776.

    Article  Google Scholar 

  • Sartorato, F., Przybylowski, L., & Sarko, D. K. (2017). Improving therapeutic outcomes in autism spectrum disorders: Enhancing social communication and sensory processing through the use of interactive robots. Journal of Psychiatric Research, 90, 1–11.

    Article  Google Scholar 

  • Saunderson, S., & Nejat, G. (2019). How robots influence humans: A survey of nonverbal communication in social human–robot interaction. International Journal of Social Robotics, 11(4), 575–608.

    Article  Google Scholar 

  • Schauß, T., Groten, R., Peer, A., & Buss, M. (2010). Evaluation of a coordinating controller for improved task performance in multi-user teleoperation. In D. Hutchison, T. Kanade, J. Kittler, J. M. Kleinberg, F. Mattern, J. C. Mitchell, M. Naor, O. Nierstrasz, C. Pandu Rangan, B. Steffen, M. Sudan, D. Terzopoulos, D. Tygar, M. Y. Vardi, G. Weikum, A. M. L. Kappers, J. B. F. van Erp, W. M. Bergmann Tiest, F. C. T. van der Helm (Eds.), Haptics: Generating and perceiving tangible sensations. Series Title: Lecture Notes in Computer Science (Vol. 6191, pp. 240–247). Springer. https://doi.org/10.1007/978-3-642-14064-8_35

  • Sheridan, T. B., & Ferrell, W. R. (1963). Remote manipulative control with transmission delay. IEEE Transactions on Human Factors in Electronics, 1, 25–29.

    Article  Google Scholar 

  • Taylor, R. M. (1990). Situational awareness rating technique (SART): The development of a tool for aircrew systems design. In Situational awareness (pp. 111–128). Routledge.

  • Tomasello, M. (2010). Origins of human communication. MIT Press.

    Google Scholar 

  • Tulli, S., Correia, F., Mascarenhas, S., Gomes, S., Melo, F. S., & Paiva, A. (2019). Effects of agents’ transparency on teamwork. In International workshop on explainable, transparent autonomous agents and multi-agent systems (pp. 22–37). Springer.

  • Uggirala, A., Gramopadhye, A. K., Melloy, B. J., & Toler, J. E. (2004). Measurement of trust in complex and dynamic systems using a quantitative approach. International Journal of Industrial Ergonomics, 34(3), 175–186.

    Article  Google Scholar 

  • Varkonyi, T. A., Rudas, I. J., Pausits, P., & Haidegger, T. (2014). Survey on the control of time delay teleoperation systems. In IEEE 18th International Conference on Intelligent Engineering Systems INES 2014 (pp. 89–94). IEEE. https://doi.org/10.1109/INES.2014.6909347

  • Watson, B., Walker, N., Ribarsky, W., & Spaulding, V. (1998). Effects of variation in system responsiveness on user performance in virtual environments. Human Factors, 40(3), 403–414.

    Article  Google Scholar 

  • Wohleber, R. W., Stowers, K., Chen, J. Y., & Barnes, M. (2017). Effects of agent transparency and communication framing on human-agent teaming. In 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 3427–3432). IEEE. https://doi.org/10.1109/SMC.2017.8123160

Download references

Acknowledgements

The work in this paper was supported by an Amazon Research Award.

Author information

Authors and Affiliations

  1. Worcester Polytechnic Institute, Worcester, MA, USA

    Jayam Patel, Prajankya Sonar & Carlo Pinciroli

Authors
  1. Jayam Patel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Prajankya Sonar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Carlo Pinciroli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carlo Pinciroli.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Patel, J., Sonar, P. & Pinciroli, C. On multi-human multi-robot remote interaction: a study of transparency, inter-human communication, and information loss in remote interaction. Swarm Intell 16, 107–142 (2022). https://doi.org/10.1007/s11721-021-00209-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11721-021-00209-2

Keywords

Search

Navigation