Journal on Multimodal User Interfaces

, Volume 11, Issue 1, pp 67–80 | Cite as

Multimodal feedback for teleoperation of multiple mobile robots in an outdoor environment

  • Ayoung Hong
  • Dong Gun Lee
  • Heinrich H. Bülthoff
  • Hyoung Il Son
Original Paper


Better situational awareness helps understand remote environments and achieve better performance in the teleoperation of multiple mobile robots (e.g., a group of unmanned aerial vehicles). Visual and force feedbacks are the most common ways of perceiving the environments accurately and effectively; however, accurate and adequate sensors for global localization are impractical in outdoor environments. Lack of this information hinders situational awareness and operating performance. In this paper, a visual and force feedback method is proposed for enhancing the situational awareness of human operators in outdoor multi-robot teleoperation. Using only the robots’ local information, the global view is fabricated from individual local views, and force feedback is determined by the velocity of individual units. The proposed feedback method is evaluated via two psychophysical experiments: maneuvering and searching tests using a human/hardware-in-the-loop system with simulated environments. In the tests, several quantitative measures are also proposed to assess the human operator’s maneuverability and situational awareness. Results of the two experiments show that the proposed multimodal feedback enhances only situational awareness of the operator.


Multimodal feedback Multi-robot systems Psychophysical evaluation Bilateral teleoperation Outdoor environment 


  1. 1.
    Murray R (2007) Recent research in cooperative control of multivehicle systems. J Dyn Syst Meas Control 129:571–583CrossRefGoogle Scholar
  2. 2.
    Lee D, Spong M (2005) Bilateral teleoperation of multiple cooperative robots over delayed communication networks: theory. In: IEEE Int Conf Robot Autom, pp 360–365Google Scholar
  3. 3.
    Franchi A, Secchi C, Son HI, Bülthoff HH, Giordano PR (2012) Bilateral teleoperation of groups of mobile robots with time-varying topology. IEEE Trans Robot 28(5):1019–1033CrossRefGoogle Scholar
  4. 4.
    Lee D, Franchi A, Son HI, Ha C, Bülthoff H, Giordano PR (2013) Semiautonomous haptic teleoperation control architecture of multiple unmanned aerial vehicles. IEEE ASME Trans Mechatron 18(4):1334–1345CrossRefGoogle Scholar
  5. 5.
    Son HI, Franchi A, Chuang LL, Kim J, Bülthoff HH, Giordano PR (2013) Human-centered design and evaluation of haptic cueing for teleoperation of multiple mobile robots. IEEE Trans Cybern 43(2):597–609CrossRefGoogle Scholar
  6. 6.
    Zhang J, Liu W, Wu Y (2011) Novel technique for vision-based UAV navigation. IEEE Trans Aerosp Electron Syst 47(4):2731–2741CrossRefGoogle Scholar
  7. 7.
    Lam T, Mulder M, Paassen v (2007) Haptic interface for UAV collision avoidance. Int J Aviat Psychol 17(2):167–195CrossRefGoogle Scholar
  8. 8.
    Son HI, Chuang L, Kim J, Bülthoff H (2011) Haptic feedback cues can improve human perceptual awareness in multi-robots teleoperation. In: Int conf on control, automation and systems, pp 1323–1328Google Scholar
  9. 9.
    Chen JY, Barnes MJ, Harper-Sciarini M (2011) Supervisory control of multiple robots: human-performance issues and user-interface design. IEEE Trans Syst Man Cybern C Appl Rev 41(4):435–454CrossRefGoogle Scholar
  10. 10.
    Labonte D, Boissy P, Michaud F (2010) Comparative analysis of 3-d robot teleoperation interfaces with novice users. IEEE Trans Syst Man Cybern B Cybern 40(5):1331–1342CrossRefGoogle Scholar
  11. 11.
    Pitman D (2010) Collaborative micro aerial vehicle exploration of outdoor environments. PhD Thesis, Massachusetts Institute of TechnologyGoogle Scholar
  12. 12.
    Giordano P, Deusch H, Lächele J, Bülthoff HH (2010) Visual-vestibular feedback for enhanced situational awareness in teleoperation of UAVs. In: AHS annual forum and technology display, pp 1–10Google Scholar
  13. 13.
    Son HI, Kim J, Chuang L, Franchi A, Robuffo Giordano P, Lee D, Bülthoff H (2011) An evaluation of haptic cues on the tele-operator’s perceptual awareness of multiple UAVs’ environments. In: World haptics conference, pp 149–154Google Scholar
  14. 14.
    Son HI, Chuang LL, Franchi A, Kim J, Lee D, Lee S.-W., Bülthoff HH, Robuffo Giordano P (2011) Measuring an operator’s maneuverability performance in the haptic teleoperation of multiple robots. In: IEEE/RSJ int conf on intelligent robots and systems, pp 3039–3046Google Scholar
  15. 15.
    Mast M, Španěl M, Arbeiter G, Štancl V, Materna Z, Weisshardt F, Burmester M, Smrž P, Graf B (2013) Teleoperation of domestic service robots: effects of global 3d environment maps in the user interface on operators’ cognitive and performance metrics. In: Social robotics. Springer, pp 392–401Google Scholar
  16. 16.
    Nielsen C, Goodrich M, Ricks R (2007) Ecological interfaces for improving mobile robot teleoperation. IEEE Trans Robot 23(5):927–941CrossRefGoogle Scholar
  17. 17.
    Kelly A, Chan N, Herman H, Huber D, Meyers R, Rander P, Warner R, Ziglar J, Capstick E (2011) Real-time photorealistic virtualized reality interface for remote mobile robot control. Int J Rob Res 30(3):384–404CrossRefGoogle Scholar
  18. 18.
    Steinfeld A, Fong T, Kaber D, Lewis M, Scholtz J, Schultz A, Goodrich M (2006) Common metrics for human-robot interaction. In: ACM SIGCHI/SIGART conf on human–robot interaction, pp 33–40Google Scholar
  19. 19.
    Saleh JA, Karray F (2011) Towards unified performance metrics for multi-robot human interaction systems. In: Autonomous and intelligent systems. Springer, pp 311–320Google Scholar
  20. 20.
    Gatsoulis Y, Virk GS, Dehghani-Sanij AA (2010) On the measurement of situation awareness for effective human–robot interaction in teleoperated systems. J Cogn Eng Decis Mak 4(1):69–98CrossRefGoogle Scholar
  21. 21.
    Lee DG, Cho GR, Lee MS, Kim B-S, Oh S, Son HI (2013) Human-centered evaluation of multi-user teleoperation for mobile manipulator in unmanned offshore plants. In: IEEE/RSJ int conf on intelligent robots and systems, pp 5431–5438Google Scholar
  22. 22.
    Schreckenghost D, Milam T, Fong T (2010) Measuring performance in real time during human-robot operations with adjustable autonomy. IEEE Intell Syst 25:36–45CrossRefGoogle Scholar
  23. 23.
    Nisky I, Mussa-Ivaldi FA, Karniel A (2013) Analytical study of perceptual and motor transparency in bilateral teleoperation. IEEE Trans Hum Mach Syst 43(6):570–581CrossRefGoogle Scholar
  24. 24.
    Glas DF, Kanda T, Ishiguro H, Hagita N (2012) Teleoperation of multiple social robots. IEEE Trans Syst Man Cybern A Syst Hum 42(3):530–544CrossRefGoogle Scholar
  25. 25.
    Franchi A, Masone C, Grabe V, Ryll M, Bülthoff HH, Giordano PR (2012) Modeling and control of UAV bearing formations with bilateral high-level steering. Int J Rob Res 31(12):1504–1525CrossRefGoogle Scholar
  26. 26.
    Hong A, Bülthoff HH, Son HI (2013) A visual and force feedback for multi-robot teleoperation in outdoor environments: a preliminary result. In: IEEE int conf robot autom, pp 1471–1478Google Scholar
  27. 27.
    Franchi A, Masone C, Bülthoff HH, Giordano PR (2011) Bilateral teleoperation of multiple uavs with decentralized bearing-only formation control. In: IEEE/RSJ int conf on intelligent robots and systems. IEEE, pp 2215–2222Google Scholar
  28. 28.
    Ryu J, Kim Y, Hannaford B (2004) Sampled-and continuous-time passivity and stability of virtual environments. IEEE Trans Robot 20(4):772–776CrossRefGoogle Scholar
  29. 29.
    Lee D, Huang K (2010) Passive-set-position-modulation framework for interactive robotic systems. IEEE Trans Robot 26(2):354–369CrossRefGoogle Scholar
  30. 30.
    Franken M, Stramigioli S, Misra S, Secchi C, Macchelli A (2011) Bilateral telemanipulation with time delays: a two-layer approach combining passivity and transparency. IEEE Trans Robot 27(4):741–756CrossRefGoogle Scholar
  31. 31.
    Experiment Database, Max Planck Campus Tübingen.
  32. 32.
    Lächele J (2011) Development of a real-time simulation environment for multiple robot systems. Diploma Thesis, Eberhard Karls Universität TübingenGoogle Scholar
  33. 33.
    Gescheider GA (1997) Psychophysics: the fundamentals. Lawrence Erlbaum Associates, Mahwah, New JerseyGoogle Scholar
  34. 34.
    Velagapudi P, Owens S, Scerri P, Sycara K, Le M (2009) Environmental factors affecting situation awareness in unmanned aerial vehicles. In: AIAA conf, pp 1–7Google Scholar

Copyright information

© SIP 2016

Authors and Affiliations

  • Ayoung Hong
    • 1
  • Dong Gun Lee
    • 2
  • Heinrich H. Bülthoff
    • 3
    • 4
  • Hyoung Il Son
    • 5
  1. 1.Institute of Robotics and Intelligent SystemsETH ZurichZurichSwitzerland
  2. 2.Institute of Industrial TechnologySamsung Heavy IndustriesDaejeonKorea
  3. 3.Max Planck Institute for Biological CyberneticsTübingenGermany
  4. 4.Department of Brain and Cognitive EngineeringKorea UniversitySeoulKorea
  5. 5.Department of Rural and Biosystems EngineeringChonnam National UniversityGwangjuRepublic of Korea

Personalised recommendations