Skip to main content
SpringerLink
Log in

Multimodal feedback for teleoperation of multiple mobile robots in an outdoor environment

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Better situational awareness helps understand remote environments and achieve better performance in the teleoperation of multiple mobile robots (e.g., a group of unmanned aerial vehicles). Visual and force feedbacks are the most common ways of perceiving the environments accurately and effectively; however, accurate and adequate sensors for global localization are impractical in outdoor environments. Lack of this information hinders situational awareness and operating performance. In this paper, a visual and force feedback method is proposed for enhancing the situational awareness of human operators in outdoor multi-robot teleoperation. Using only the robots’ local information, the global view is fabricated from individual local views, and force feedback is determined by the velocity of individual units. The proposed feedback method is evaluated via two psychophysical experiments: maneuvering and searching tests using a human/hardware-in-the-loop system with simulated environments. In the tests, several quantitative measures are also proposed to assess the human operator’s maneuverability and situational awareness. Results of the two experiments show that the proposed multimodal feedback enhances only situational awareness of the operator.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Murray R (2007) Recent research in cooperative control of multivehicle systems. J Dyn Syst Meas Control 129:571–583

    Article  Google Scholar 

  2. Lee D, Spong M (2005) Bilateral teleoperation of multiple cooperative robots over delayed communication networks: theory. In: IEEE Int Conf Robot Autom, pp 360–365

  3. Franchi A, Secchi C, Son HI, Bülthoff HH, Giordano PR (2012) Bilateral teleoperation of groups of mobile robots with time-varying topology. IEEE Trans Robot 28(5):1019–1033

    Article  Google Scholar 

  4. Lee D, Franchi A, Son HI, Ha C, Bülthoff H, Giordano PR (2013) Semiautonomous haptic teleoperation control architecture of multiple unmanned aerial vehicles. IEEE ASME Trans Mechatron 18(4):1334–1345

    Article  Google Scholar 

  5. Son HI, Franchi A, Chuang LL, Kim J, Bülthoff HH, Giordano PR (2013) Human-centered design and evaluation of haptic cueing for teleoperation of multiple mobile robots. IEEE Trans Cybern 43(2):597–609

    Article  Google Scholar 

  6. Zhang J, Liu W, Wu Y (2011) Novel technique for vision-based UAV navigation. IEEE Trans Aerosp Electron Syst 47(4):2731–2741

    Article  Google Scholar 

  7. Lam T, Mulder M, Paassen v (2007) Haptic interface for UAV collision avoidance. Int J Aviat Psychol 17(2):167–195

    Article  Google Scholar 

  8. Son HI, Chuang L, Kim J, Bülthoff H (2011) Haptic feedback cues can improve human perceptual awareness in multi-robots teleoperation. In: Int conf on control, automation and systems, pp 1323–1328

  9. Chen JY, Barnes MJ, Harper-Sciarini M (2011) Supervisory control of multiple robots: human-performance issues and user-interface design. IEEE Trans Syst Man Cybern C Appl Rev 41(4):435–454

    Article  Google Scholar 

  10. Labonte D, Boissy P, Michaud F (2010) Comparative analysis of 3-d robot teleoperation interfaces with novice users. IEEE Trans Syst Man Cybern B Cybern 40(5):1331–1342

    Article  Google Scholar 

  11. Pitman D (2010) Collaborative micro aerial vehicle exploration of outdoor environments. PhD Thesis, Massachusetts Institute of Technology

  12. Giordano P, Deusch H, Lächele J, Bülthoff HH (2010) Visual-vestibular feedback for enhanced situational awareness in teleoperation of UAVs. In: AHS annual forum and technology display, pp 1–10

  13. Son HI, Kim J, Chuang L, Franchi A, Robuffo Giordano P, Lee D, Bülthoff H (2011) An evaluation of haptic cues on the tele-operator’s perceptual awareness of multiple UAVs’ environments. In: World haptics conference, pp 149–154

  14. Son HI, Chuang LL, Franchi A, Kim J, Lee D, Lee S.-W., Bülthoff HH, Robuffo Giordano P (2011) Measuring an operator’s maneuverability performance in the haptic teleoperation of multiple robots. In: IEEE/RSJ int conf on intelligent robots and systems, pp 3039–3046

  15. Mast M, Španěl M, Arbeiter G, Štancl V, Materna Z, Weisshardt F, Burmester M, Smrž P, Graf B (2013) Teleoperation of domestic service robots: effects of global 3d environment maps in the user interface on operators’ cognitive and performance metrics. In: Social robotics. Springer, pp 392–401

  16. Nielsen C, Goodrich M, Ricks R (2007) Ecological interfaces for improving mobile robot teleoperation. IEEE Trans Robot 23(5):927–941

    Article  Google Scholar 

  17. Kelly A, Chan N, Herman H, Huber D, Meyers R, Rander P, Warner R, Ziglar J, Capstick E (2011) Real-time photorealistic virtualized reality interface for remote mobile robot control. Int J Rob Res 30(3):384–404

    Article  Google Scholar 

  18. Steinfeld A, Fong T, Kaber D, Lewis M, Scholtz J, Schultz A, Goodrich M (2006) Common metrics for human-robot interaction. In: ACM SIGCHI/SIGART conf on human–robot interaction, pp 33–40

  19. Saleh JA, Karray F (2011) Towards unified performance metrics for multi-robot human interaction systems. In: Autonomous and intelligent systems. Springer, pp 311–320

  20. Gatsoulis Y, Virk GS, Dehghani-Sanij AA (2010) On the measurement of situation awareness for effective human–robot interaction in teleoperated systems. J Cogn Eng Decis Mak 4(1):69–98

    Article  Google Scholar 

  21. Lee DG, Cho GR, Lee MS, Kim B-S, Oh S, Son HI (2013) Human-centered evaluation of multi-user teleoperation for mobile manipulator in unmanned offshore plants. In: IEEE/RSJ int conf on intelligent robots and systems, pp 5431–5438

  22. Schreckenghost D, Milam T, Fong T (2010) Measuring performance in real time during human-robot operations with adjustable autonomy. IEEE Intell Syst 25:36–45

    Article  Google Scholar 

  23. Nisky I, Mussa-Ivaldi FA, Karniel A (2013) Analytical study of perceptual and motor transparency in bilateral teleoperation. IEEE Trans Hum Mach Syst 43(6):570–581

    Article  Google Scholar 

  24. Glas DF, Kanda T, Ishiguro H, Hagita N (2012) Teleoperation of multiple social robots. IEEE Trans Syst Man Cybern A Syst Hum 42(3):530–544

    Article  Google Scholar 

  25. Franchi A, Masone C, Grabe V, Ryll M, Bülthoff HH, Giordano PR (2012) Modeling and control of UAV bearing formations with bilateral high-level steering. Int J Rob Res 31(12):1504–1525

    Article  Google Scholar 

  26. Hong A, Bülthoff HH, Son HI (2013) A visual and force feedback for multi-robot teleoperation in outdoor environments: a preliminary result. In: IEEE int conf robot autom, pp 1471–1478

  27. Franchi A, Masone C, Bülthoff HH, Giordano PR (2011) Bilateral teleoperation of multiple uavs with decentralized bearing-only formation control. In: IEEE/RSJ int conf on intelligent robots and systems. IEEE, pp 2215–2222

  28. Ryu J, Kim Y, Hannaford B (2004) Sampled-and continuous-time passivity and stability of virtual environments. IEEE Trans Robot 20(4):772–776

    Article  Google Scholar 

  29. Lee D, Huang K (2010) Passive-set-position-modulation framework for interactive robotic systems. IEEE Trans Robot 26(2):354–369

    Article  Google Scholar 

  30. Franken M, Stramigioli S, Misra S, Secchi C, Macchelli A (2011) Bilateral telemanipulation with time delays: a two-layer approach combining passivity and transparency. IEEE Trans Robot 27(4):741–756

    Article  Google Scholar 

  31. Experiment Database, Max Planck Campus Tübingen. https://experiments.tuebingen.mpg.de/

  32. Lächele J (2011) Development of a real-time simulation environment for multiple robot systems. Diploma Thesis, Eberhard Karls Universität Tübingen

  33. Gescheider GA (1997) Psychophysics: the fundamentals. Lawrence Erlbaum Associates, Mahwah, New Jersey

    Google Scholar 

  34. Velagapudi P, Owens S, Scerri P, Sycara K, Le M (2009) Environmental factors affecting situation awareness in unmanned aerial vehicles. In: AIAA conf, pp 1–7

Download references

Acknowledgments

This research was supported in part by the Max Planck Society, in part by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future Planning under Grant NRF-2015R1C1A1A02036875, in part by Chonnam National University (2015-0536), and in part by a Grant (115062-2) funded by the Ministry of Agriculture, Food and Rural Affairs (MAFRA, Korea).

Author information

Authors and Affiliations

  1. Institute of Robotics and Intelligent Systems, ETH Zurich, 8092, Zurich, Switzerland

    Ayoung Hong

  2. Institute of Industrial Technology, Samsung Heavy Industries, Daejeon, 305-380, Korea

    Dong Gun Lee

  3. Max Planck Institute for Biological Cybernetics, Spemannstraße 38, 72076, Tübingen, Germany

    Heinrich H. Bülthoff

  4. Department of Brain and Cognitive Engineering, Korea University, Anam-dong, Seongbuk-gu, Seoul, 136-713, Korea

    Heinrich H. Bülthoff

  5. Department of Rural and Biosystems Engineering, Chonnam National University, 77 Yongbong-ro, Buk-gu, Gwangju, 550-757, Republic of Korea

    Hyoung Il Son

Authors
  1. Ayoung Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dong Gun Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Heinrich H. Bülthoff
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hyoung Il Son
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hyoung Il Son.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hong, A., Lee, D.G., Bülthoff, H.H. et al. Multimodal feedback for teleoperation of multiple mobile robots in an outdoor environment. J Multimodal User Interfaces 11, 67–80 (2017). https://doi.org/10.1007/s12193-016-0230-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-016-0230-y

Keywords

Search

Navigation