Journal on Multimodal User Interfaces

, Volume 11, Issue 1, pp 67–80

Multimodal feedback for teleoperation of multiple mobile robots in an outdoor environment

  • Ayoung Hong
  • Dong Gun Lee
  • Heinrich H. Bülthoff
  • Hyoung Il Son
Original Paper

DOI: 10.1007/s12193-016-0230-y

Cite this article as:
Hong, A., Lee, D.G., Bülthoff, H.H. et al. J Multimodal User Interfaces (2017) 11: 67. doi:10.1007/s12193-016-0230-y


Better situational awareness helps understand remote environments and achieve better performance in the teleoperation of multiple mobile robots (e.g., a group of unmanned aerial vehicles). Visual and force feedbacks are the most common ways of perceiving the environments accurately and effectively; however, accurate and adequate sensors for global localization are impractical in outdoor environments. Lack of this information hinders situational awareness and operating performance. In this paper, a visual and force feedback method is proposed for enhancing the situational awareness of human operators in outdoor multi-robot teleoperation. Using only the robots’ local information, the global view is fabricated from individual local views, and force feedback is determined by the velocity of individual units. The proposed feedback method is evaluated via two psychophysical experiments: maneuvering and searching tests using a human/hardware-in-the-loop system with simulated environments. In the tests, several quantitative measures are also proposed to assess the human operator’s maneuverability and situational awareness. Results of the two experiments show that the proposed multimodal feedback enhances only situational awareness of the operator.


Multimodal feedback Multi-robot systems Psychophysical evaluation Bilateral teleoperation Outdoor environment 

Funding information

Funder NameGrant NumberFunding Note
Max-Planck-Gesellschaft (DE)
    National Research Foundation of Korea (KR)
    • NRF-2015R1C1A1A02036875
    Chonnam National University
    • 2015-0536
    Ministry of Agriculture, Food and Rural Affairs
    • 115062-2

    Copyright information

    © SIP 2016

    Authors and Affiliations

    • Ayoung Hong
      • 1
    • Dong Gun Lee
      • 2
    • Heinrich H. Bülthoff
      • 3
      • 4
    • Hyoung Il Son
      • 5
    1. 1.Institute of Robotics and Intelligent SystemsETH ZurichZurichSwitzerland
    2. 2.Institute of Industrial TechnologySamsung Heavy IndustriesDaejeonKorea
    3. 3.Max Planck Institute for Biological CyberneticsTübingenGermany
    4. 4.Department of Brain and Cognitive EngineeringKorea UniversitySeoulKorea
    5. 5.Department of Rural and Biosystems EngineeringChonnam National UniversityGwangjuRepublic of Korea

    Personalised recommendations