Multi-robot Systems, Virtual Reality and ROS: Developing a New Generation of Operator Interfaces

  • Juan Jesús Roldán
  • Elena Peña-Tapia
  • David Garzón-Ramos
  • Jorge de León
  • Mario Garzón
  • Jaime del Cerro
  • Antonio Barrientos
Part of the Studies in Computational Intelligence book series (SCI, volume 778)


This chapter describes a series of works developed in order to integrate ROS-based robots with Unity-based virtual reality interfaces. The main goal of this integration is to develop immersive monitoring and commanding interfaces, able to improve the operator’s situational awareness without increasing its workload. In order to achieve this, the available technologies and resources are analyzed and multiple ROS packages and Unity assets are applied, such as \(multimaster\_fkie\), \(rosbridge\_suite\), RosBridgeLib and SteamVR. Moreover, three applications are presented: an interface for monitoring a fleet of drones, another interface for commanding a robot manipulator and an integration of multiple ground and aerial robots. Finally, some experiences and lessons learned, useful for future developments, are reported.


Multi-robot systems Virtual reality Operator interfaces Immersive teleoperation 



This work is framed on SAVIER (Situational Awareness Virtual EnviRonment) Project, which is both supported and funded by Airbus Defence and Space. The research leading to these results has received funding from the RoboCity2030-III-CM project (Robótica aplicada a la mejora de la calidad de vida de los ciudadanos. Fase III; S2013/MIT-2748), funded by Programas de Actividades I\(+\)D en la Comunidad de Madrid and cofunded by Structural Funds of the EU, and from the DPI2014-56985-R project (Protección robotizada de infraestructuras críticas) funded by the Ministerio de Economía y Competitividad of Gobierno de España.


  1. 1.
    M. Garzón, J. Valente, J.J. Roldán, L. Cancar, A. Barrientos, J. Del Cerro, A multirobot system for distributed area coverage and signal searching in large outdoor scenarios. J. Field Robot. 33(8), 1087–1106 (2016)CrossRefGoogle Scholar
  2. 2.
    J.J. Roldán, P. Garcia-Aunon, M. Garzón, J. de León, J. del Cerro, A. Barrientos, Heterogeneous multi-robot system for mapping environmental variables of greenhouses. Sensors 16(7), 1018 (2016)CrossRefGoogle Scholar
  3. 3.
    M. Garzón, J. Valente, D. Zapata, A. Barrientos, An aerial-ground robotic system for navigation and obstacle mapping in large outdoor areas. Sensors 13(1), 1247–1267 (2013)CrossRefGoogle Scholar
  4. 4.
    J.Y. Chen, E.C. Haas, M.J. Barnes, Human performance issues and user interface design for teleoperated robots. IEEE Trans. Syst. Man Cybern. Part C (Applications and Reviews) 37(6), 1231–1245 (2007)CrossRefGoogle Scholar
  5. 5.
    J. Ruiz, A. Viguria, J. Martinez-de Dios, A. Ollero, Immersive displays for building spatial knowledge in multi-uav operations, in 2015 International Conference on Unmanned Aircraft Systems (ICUAS) (IEEE, 2015), pp. 1043–1048Google Scholar
  6. 6.
    J.T. Hansberger, Development of the next generation of adaptive interfaces. Technical report, DTIC Document (2015)Google Scholar
  7. 7.
    J.J. Roldán, M.A. Olivares-Méndez, J. del Cerro, A. Barrientos, Analyzing and improving multi-robot missions by means of process mining. Auton. Robot. 1(1), 1–21 (2017)Google Scholar
  8. 8.
    P. Ulam, Y. Endo, A. Wagner, R. Arkin, Integrated mission specication and task allocation for robot teams-part 2: Testing and evaluation. Technical report, GEORGIA INST OF TECH ATLANTA COLL OF COMPUTING (2006)Google Scholar
  9. 9.
    S. Tully, G. Kantor, H. Choset, Leap-frog path design for multi-robot cooperative localization, in Field and Service Robotics (Springer, Berlin, 2010), pp. 307–317Google Scholar
  10. 10.
    A. Janchiv, D. Batsaikhan, G. hwan Kim, S.G. Lee, Complete coverage path planning for multi-robots based on, in 2011 11th International Conference on Control, Automation and Systems (ICCAS) (IEEE, 2011), pp. 824–827Google Scholar
  11. 11.
    M. Lindemuth, R. Murphy, E. Steimle, W. Armitage, K. Dreger, T. Elliot, M. Hall, D. Kalyadin, J. Kramer, M. Palankar et al., Sea robot-assisted inspection. IEEE Robot. Autom. Mag. 18(2), 96–107 (2011)CrossRefGoogle Scholar
  12. 12.
    J. Valente, D. Sanz, A. Barrientos, Jd Cerro, Á. Ribeiro, C. Rossi, An air-ground wireless sensor network for crop monitoring. Sensors 11(6), 6088–6108 (2011)CrossRefGoogle Scholar
  13. 13.
    N.A. Tsokas, K.J. Kyriakopoulos, Multi-robot multiple hypothesis tracking for pedestrian tracking. Auton. Robot. 32(1), 63–79 (2012)CrossRefGoogle Scholar
  14. 14.
    L. Cantelli, M. Mangiameli, C.D. Melita, G. Muscato, Uav/ugv cooperation for surveying operations in humanitarian demining, in 2013 IEEE International symposium on Safety, Security, and Rescue Robotics (SSRR) (IEEE, 2013), pp. 1–6Google Scholar
  15. 15.
    G. De Cubber, D. Doroftei, D. Serrano, K. Chintamani, R. Sabino, S. Ourevitch, The eu-icarus project: developing assistive robotic tools for search and rescue operations, in 2013 IEEE international symposium on Safety, Security, and Rescue Robotics (SSRR) (IEEE, 2013), pp. 1–4Google Scholar
  16. 16.
    I. Kruijff-Korbayová, F. Colas, M. Gianni, F. Pirri, J. Greeff, K. Hindriks, M. Neerincx, P. Ögren, T. Svoboda, R. Worst, Tradr project: Long-term human-robot teaming for robot assisted disaster response. KI-Künstliche Intelligenz 29(2), 193–201 (2015)CrossRefGoogle Scholar
  17. 17.
    J. Gregory, J. Fink, E. Stump, J. Twigg, J. Rogers, D. Baran, N. Fung, S. Young, Application of multi-robot systems to disaster-relief scenarios with limited communication, in Field and Service Robotics (Springer, Berlin, 2016), pp. 639–653Google Scholar
  18. 18.
    A.C. Kapoutsis, S.A. Chatzichristofis, L. Doitsidis, J.B. de Sousa, J. Pinto, J. Braga, E.B. Kosmatopoulos, Real-time adaptive multi-robot exploration with application to underwater map construction. Auton. Robot. 40(6), 987–1015 (2016)CrossRefGoogle Scholar
  19. 19.
    C. Lesire, G. Infantes, T. Gateau, M. Barbier, A distributed architecture for supervision of autonomous multi-robot missions. Auton. Robot. 40(7), 1343–1362 (2016)CrossRefGoogle Scholar
  20. 20.
    N. Agmon, O. Maximov, A. Rosenfeld, S. Shlomai, S. Kraus, Multiple robots for multiple missions: architecture for complex collaborationGoogle Scholar
  21. 21.
    X.J. Yang, C.D. Wickens, K. Hölttä-Otto, How users adjust trust in automation: Contrast effect and hindsight bias, in Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 60 (SAGE Publications Sage CA: Los Angeles, CA, 2016), pp. 196–200CrossRefGoogle Scholar
  22. 22.
    C. Sampedro, H. Bavle, J.L. Sanchez-Lopez, R.A.S. Fernández, A. Rodríguez-Ramos, M. Molina, P. Campoy, A flexible and dynamic mission planning architecture for uav swarm coordination, in 2016 International Conference on Unmanned Aircraft Systems (ICUAS) (IEEE, 2016), pp. 355–363Google Scholar
  23. 23.
    T. Nestmeyer, P.R. Giordano, H.H. Bülthoff, A. Franchi, Decentralized simultaneous multi-target exploration using a connected network of multiple robots. Auton. Robot. 41(4), 989–1011 (2017)CrossRefGoogle Scholar
  24. 24.
    M. Garzón, J. Valente, J.J. Roldán, D. Garzón-Ramos, J. de León, A. Barrientos, J. del Cerro, Using ros in multi-robot systems: Experiences and lessons learned from real-world field tests, in Robot Operating System (ROS) (Springer, Berlin, 2017), pp. 449–483Google Scholar
  25. 25.
    J.J. Roldán, E. Peña-Tapia, A. Martín-Barrio, M.A. Olivares-Méndez, J. Del Cerro, A. Barrientos, Multi-robot interfaces and operator situational awareness: study of the impact of immersion and prediction. Sensors 17(8), 1720 (2017)CrossRefGoogle Scholar
  26. 26.
    M.R. Endsley, Design and evaluation for situation awareness enhancement, in Proceedings of the human factors and ergonomics society annual meeting, vol. 32 (SAGE Publications, 1988), pp. 97–101CrossRefGoogle Scholar
  27. 27.
    J.L. Drury, J. Scholtz, H.A. Yanco, Awareness in human-robot interactions, in IEEE International Conference on Systems, Man and Cybernetics, vol. 1 (IEEE, 2003), pp. 912–918Google Scholar
  28. 28.
    M.R. Endsley, Situation awareness global assessment technique (sagat), in Proceedings of the IEEE National Aerospace and Electronics Conference. NAECON (IEEE, 1988), pp. 789–795Google Scholar
  29. 29.
    P. Salmon, N. Stanton, G. Walker, D. Green, Situation awareness measurement: a review of applicability for c4i environments. Appl. Ergon. 37(2), 225–238 (2006)CrossRefGoogle Scholar
  30. 30.
    J. Scholtz, J. Young, J.L. Drury, H.A. Yanco, Evaluation of human-robot interaction awareness in search and rescue, in Proceedings of the ICRA’04 IEEE International Conference on Robotics and Automation, vol. 3 (IEEE, 2004), pp. 2327–2332Google Scholar
  31. 31.
    N. Li, S. Cartwright, A. Shekhar Nittala, E. Sharlin, M. Costa Sousa, Flying frustum: a spatial interface for enhancing human-uav awareness, in Proceedings of the 3rd International Conference on Human-Agent Interaction (ACM, 2015), pp. 27–31Google Scholar
  32. 32.
    R.J. Lysaght, S.G. Hill, A. Dick, B.D. Plamondon, P.M. Linton, Operator workload: Comprehensive review and evaluation of operator workload methodologies. Technical report, DTIC Document (1989)Google Scholar
  33. 33.
    N. Moray, Mental Workload: Its Theory and Measurement, vol. 8 (Springer Science & Business Media, 2013)Google Scholar
  34. 34.
    S.G. Hart, L.E. Staveland, Development of nasa-tlx (task load index): Results of empirical and theoretical research. Adv. Psychol. 52, 139–183 (1988)CrossRefGoogle Scholar
  35. 35.
    S.R. Dixon, C.D. Wickens, D. Chang, Mission control of multiple unmanned aerial vehicles: a workload analysis. Hum. Factors J. Hum. Factors Ergon. Soc. 47(3), 479–487 (2005)CrossRefGoogle Scholar
  36. 36.
    B. Jacobs, E. De Visser, A. Freedy, P. Scerri, Application of Intelligent Aiding to Enable Single Operator Multiple uav Supervisory Control, Association for the advancement of artificial intelligence (Palo Alto, CA, 2010)Google Scholar
  37. 37.
    M.L. Cummings, C. Mastracchio, K.M. Thornburg, A. Mkrtchyan, Boredom and distraction in multiple unmanned vehicle supervisory control. Interact. Comput. 25(1), 34–47 (2013)CrossRefGoogle Scholar
  38. 38.
    D. McDuff, S. Gontarek, R. Picard, Remote measurement of cognitive stress via heart rate variability, in 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (IEEE, 2014), pp. 2957–2960Google Scholar
  39. 39.
    H. Kurniawan, A.V. Maslov, M. Pechenizkiy, Stress detection from speech and galvanic skin response signals, in IEEE 26th International Symposium on Computer-Based Medical Systems (CBMS) (IEEE, 2013), pp. 209–214Google Scholar
  40. 40.
    E.A. Kirchner, S.K. Kim, M. Tabie, H. Wöhrle, M. Maurus, F. Kirchner, An intelligent man-machine interfacemulti-robot control adapted for task engagement based on single-trial detectability of p300. Front. Hum. Neurosci. 10(2016)Google Scholar
  41. 41.
    R. Parasuraman, T.B. Sheridan, C.D. Wickens, A model for types and levels of human interaction with automation. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 30(3), 286–297 (2000)CrossRefGoogle Scholar
  42. 42.
    B.D. Simpson, R.S. Bolia, M.H. Draper, Spatial audio display concepts supporting situation awareness for operators of unmanned aerial vehicles, Human Performance, Situation Awareness, and Automation: Current Research and Trends HPSAA II, Volumes I and II, vol. 2 (2013), p. 61Google Scholar
  43. 43.
    S. Scheggi, M. Aggravi, F. Morbidi, D. Prattichizzo, Cooperative human-robot haptic navigation, in IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2014), pp. 2693–2698Google Scholar
  44. 44.
    C.E. Lathan, M. Tracey, The effects of operator spatial perception and sensory feedback on human-robot teleoperation performance. Presence Teleoper. Virtual Environ. 11(4), 368–377 (2002)CrossRefGoogle Scholar
  45. 45.
    V.M. Monajjemi, S. Pourmehr, S.A. Sadat, F. Zhan, J. Wawerla, G. Mori, R. Vaughan, Integrating multi-modal interfaces to command uavs, in Proceedings of the 2014 ACM/IEEE International Conference on Human-Robot Interaction (ACM, 2014), pp. 106–106Google Scholar
  46. 46.
    S. Kavitha, S. Veena, R. Kumaraswamy, Development of automatic speech recognition system for voice activated ground control system, in International Conference on Trends in Automation, Communications and Computing Technology (I-TACT-15), vol. 1 (IEEE, 2015), pp. 1–5Google Scholar
  47. 47.
    T. Mantecón, C.R. del Blanco, F. Jaureguizar, N. García, New generation of human machine interfaces for controlling uav through depth-based gesture recognition, in SPIE Defense+ Security, International Society for Optics and Photonics (2014), pp. 90840C–90840CGoogle Scholar
  48. 48.
    J. Nagi, A. Giusti, G.A. Di Caro, L.M. Gambardella, Human control of uavs using face pose estimates and hand gestures, in Proceedings of the 2014 ACM/IEEE International Conference on Human-Robot Interaction, (ACM, 2014), pp. 252–253Google Scholar
  49. 49.
    M. Hou, H. Zhu, M. Zhou, G.R. Arrabito, Optimizing operator–agent interaction in intelligent adaptive interface design: a conceptual framework. IEEE Trans. Syst. Man Cybern. Part C (Applications and Reviews) 41(2), 161–178 (2011)Google Scholar
  50. 50.
    J.L. Drury, J. Richer, N. Rackliffe, M.A. Goodrich, Comparing situation awareness for two unmanned aerial vehicle human interface approaches. Technical report, Mitre Corp Bedford MA (2006)Google Scholar
  51. 51.
    K. Foit, Mixed reality as a tool supporting programming of the robot, in Advanced Materials Research, vol. 1036 (Trans Tech Publ, 2014), pp. 737–742CrossRefGoogle Scholar
  52. 52.
    D.C. Niehorster, L. Li, M. Lappe, The accuracy and precision of position and orientation tracking in the htc vive virtual reality system for scientific research. i-Perception 8(3), 2041669517708205 (2017)Google Scholar
  53. 53.
    F. Navarro, J. Fdez, M. Garzon, J.J. Roldán, A. Barrientos, Integrating 3d reconstruction and virtual reality: a new approach for immersive teleoperation, in Robot 2017: Third Iberian Robotics Conference (Springer, 2018), pp. X–YGoogle Scholar
  54. 54.
    S.H. Juan, F.H. Cotarelo, Multi-Master Ros Systems, Institut de robotics and industrial informatics (2015)Google Scholar
  55. 55.
    C. Crick, G. Jay, S. Osentoski, B. Pitzer, O.C. Jenkins, Rosbridge: Ros for non-ros users, in Robotics Research (Springer, Berlin, 2017), pp. 493–504Google Scholar
  56. 56.
    P. Codd-Downey, A.S.H.W. Mojiri Forooshani, M. Jenkin, From ros to unity: leveraging robot and virtual environment middleware for immersive teleoperation (2014)Google Scholar
  57. 57.
    E. Peña-Tapia, J.J. Roldán Gómez, M. Garzón, A. Martín-Barrio, A. Barrientos-Cruz, Interfaz de control para un robot manipulador mediante realidad virtual (2017)Google Scholar
  58. 58.
    I.A. Sucan, S. Chitta, Moveit!. (2013)
  59. 59.
    I.A. Şucan, M. Moll, L.E. Kavraki, The open motion planning library. IEEE Robot. Autom. Mag. 19(4), 72–82 (2012)CrossRefGoogle Scholar
  60. 60.
    J.J. Kuffner, S.M. LaValle, Rrt-connect: an efficient approach to single-query path planning, in Proceedings of the ICRA’00 IEEE International Conference on Robotics and Automation, vol. 2 (IEEE, 2000), pp. 995–1001Google Scholar
  61. 61.
    R. Guzman, R. Navarro, M. Beneto, D. Carbonell, Robotnikprofessional service robotics applications with ros, in Robot Operating System (ROS) (Springer, Berlin, 2016), pp. 253–288Google Scholar
  62. 62.
    R. Guzmán, R. Navarro, M. Cantero, J. Ariño, Robotnikprofessional service robotics applications with ros (2), in Robot Operating System (ROS) (Springer, Berlin, 2017), pp. 419–447Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2019

Authors and Affiliations

  • Juan Jesús Roldán
    • 1
  • Elena Peña-Tapia
    • 1
  • David Garzón-Ramos
    • 2
  • Jorge de León
    • 1
  • Mario Garzón
    • 1
  • Jaime del Cerro
    • 1
  • Antonio Barrientos
    • 1
  1. 1.Centro de Automática y Robótica (UPM-CSIC)Universidad Politécnica de MadridMadridSpain
  2. 2.IRIDIA, Université Libre de Bruxelles (ULB)BrusselsBelgium

Personalised recommendations