Skip to main content
SpringerLink
Log in

Haptic teleoperation of a multirotor aerial robot using path planning with human intention estimation

  • Original Research Paper
  • Published:
Intelligent Service Robotics Aims and scope Submit manuscript

Abstract

Classical haptic teleoperation systems heavily rely on operators’ intelligence and efforts in aerial robot navigation tasks, thereby posing significantly users’ workloads. In this paper, a novel shared control scheme is presented facilitating a multirotor aerial robot haptic teleoperation system that exhibits autonomous navigation capability. A hidden Markov model filter is proposed to identify the intention state of operator based on human inputs from haptic master device, which is subsequently adopted to derive goal position for a heuristic sampling based local path planner. The human inputs are considered as commanded velocity for a trajectory servo controller to drive the robot along the planned path. In addition, vehicle velocity is perceived by the user via haptic feedback on master device to enhance situation awareness and navigation safety of the user. An experimental study was conducted in a simulated and a physical environment, and the results verify the effectiveness of the novel scheme in safe navigation of aerial robots. A user study was carried out between a classical haptic teleoperation system and the proposed approach in the identical simulated complex environment. The flight data and task load index (TLX) are acquired and analyzed. Compared with the conventional haptic teleoperation scheme, the proposed scheme exhibits superior performance in safe and fast navigation of the multirotor vehicle, and is also of low task and cognitive loads.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Goertz RC (1954) Mechanical master-slave manipulator. Nucleonics(US) Ceased publication 12(11):45–46

    Google Scholar 

  2. Clement G, Fournier R, Gravez P, Morillon J (1988) Computer aided teleoperation: from arm to vehicle control. In: 1988 IEEE International Conference on Robotics and Automation(ICRA), IEEE, pp 590–592

  3. Hong SG, Lee JJ, Kim S (1999) Generating artificial force for feedback control of teleoperated mobile robots. In: 1999 IEEE/RSJ international conference intelligent robots and systems, IEEE, pp 1721–1726

  4. Li X, Song A, Li H, Lu W, Mao C (2012) Real-time obstacle avoidance for telerobotic systems based on equipotential surface. Int. J. Adv. Robot. Syst. 9(3):71–79

    Article  Google Scholar 

  5. Luo J, Lin Z, Li Y, Yang C (2020) A teleoperation framework for mobile robots based on shared control. IEEE Robot Autom Lett 5(2):377–384

    Article  Google Scholar 

  6. Hong SG, Kim BS, Kim S, Lee JJ (1998) Artificial force reflection control for teleoperated mobile robots. Mechatronics 8(6):707–717

    Article  Google Scholar 

  7. Elhajj I, Xi N, Fung WK, Liu YH, Li WJ, Kaga T, Fukuda T (2001) Haptic information in internet-based teleoperation. IEEE Trans Mechatron 6(3):295–304

    Article  Google Scholar 

  8. Lee S, Sukhatme GS, Kim GJ, Park CM (2002) Haptic control of a mobile robot: a user study. In: 2002 IEEE/RSJ international conference intelligent robots and systems, IEEE, vol 3, pp 2867–2874

  9. Diolaiti N, Melchiorri C (2002) Teleoperation of a mobile robot through haptic feedback. In: 2002 IEEE international workshop haptic virtual environments and their applications, IEEE, pp 67–72

  10. Lam TM, Boschloo HW, Mulder M, van Paassen MM (2009) Artificial force field for haptic feedback in uav teleoperation. IEEE Trans Syst Man Cybern Part A Syst Hum 39(6):1316–1330

    Article  Google Scholar 

  11. Franchi A, Secchi C, Son HI, Bulthoff HH, Giordano PR (2012) Bilateral teleoperation of groups of mobile robots with time-varying topology. IEEE Trans Robot Autom 28(5):1019–1033

    Article  Google Scholar 

  12. Lee D, Franchi A, Son HI, Ha C, Bulthoff HH, Giordano PR (2013) Semiautonomous haptic teleoperation control architecture of multiple unmanned aerial vehicles. IEEE Trans Mechatron 18(4):1334–1345

    Article  Google Scholar 

  13. Mersha AY, Stramigioli S, Carloni R (2014) On bilateral teleoperation of aerial robots. IEEE Trans Robot 30(1):258–274

    Article  Google Scholar 

  14. Hou X, Mahony R (2016) Comparative study of haptic interfaces for bilateral teleoperation of vtol aerial robots. IEEE Trans Syst Man Cybern Syst 46(10):1352–1363

    Article  Google Scholar 

  15. Li W, Ding L, Gao H, Tavakoli M (2020) Haptic tele-driving of wheeled mobile robots under nonideal wheel rolling, kinematic control and communication time delay. IEEE Trans Syst Man Cybern Syst 50(1):336–347

    Article  Google Scholar 

  16. Mahony R, Schill F, Corke P, Oh YS (2019) A new framework for force feedback teleoperation of robotic vehicles based on optical flow. In: 2009 IEEE international conference on robotics and automation(ICRA), IEEE, pp 1079–1085

  17. Omari S, Hua MD, Ducard G, Hamel T (2013) Bilateral haptic teleoperation of vtol uavs. In: 2013 IEEE international conference on robotics and automation(ICRA), IEEE, pp 2385–2391

  18. Hou X, Mahony R (2016) Dynamic kinesthetic boundary for haptic teleoperation of vtol aerial robots in complex environments. IEEE Trans Syst Man Cybern Syst 46(5):694–705

    Article  Google Scholar 

  19. Luo J, Yang CG, Wang N, Wang M (2019) Enhanced teleoperation performance using hybrid control and virtual fixture. Int J Syst Sci 50(3):451–462

    Article  MathSciNet  Google Scholar 

  20. Jiang S, Lin C, Huang K, Song K (2017) Shared control design of a walking-assistant robot. IEEE Trans Control Syst Technol 25(6):2143–2150

    Article  Google Scholar 

  21. Kong H, Yang C, Li G, Dai S (2020) A semg-based shared control system with no-target obstacle avoidance for omnidirectional mobile robots. IEEE Access 8:26030–26040

    Article  Google Scholar 

  22. Nieto J, Slawinski E, Mut V, Wagner B (2010) Mobile robot teleoperation augmented with prediction and path-planning. Anal Design Eval Hum Mach Syst 11:53–58

    Google Scholar 

  23. Nieto J, Slawinski E, Mut V, Wagner B (2012) Toward safe and stable time-delayed mobile robot teleoperation through sampling-based path planning. Robotica 30(3):351–361

    Article  Google Scholar 

  24. Sheridan TB (1989) Telerobotics. Automatica 25(4):487–507

    Article  Google Scholar 

  25. Petkovic T, Puljiz D, Markovic I, Hein B (2019) Human intention estimation based on hidden markov model motion validation for safe flexible robotized warehouses. Robot Comput Integr Manuf 57:182–196

    Article  Google Scholar 

  26. Khokar KH, Alqasemi R, Sarkar S, Dubey RV (2013) Human motion intention based scaled teleoperation for orientation assistance in preshaping for grasping. In: 2013 IEEE International Conference on Rehabilitation Robotics (ICORR), IEEE, pp 1–6

  27. Li Y, Ge SS (2014) Human-robot collaboration based on motion intention estimation. IEEE Trans Mechatron 19(3):1007–1014

    Article  Google Scholar 

  28. Yu X, He W, Li Y, Xue C, Li J, Zou J, Yang C (2019) Bayesian estimation of human impedance and motion intention for human-robot collaboration. IEEE Transactions on Cybernetics, pp 1–13

  29. Neville H (1985) Impedance control: An approach to manipulation: Part I–Theory. J Dyn Syst Meas Control, 107:1–24. Parts I, II, III

  30. Mut V, Postigo J, Slawinski E, Kuchen B (2002) Bilateral teleoperation of mobile robots. Robotica 20(02):213–221

    Article  Google Scholar 

  31. Alaimo S, Pollini L, Buelthoff HH (2011) Admittance-based bilateral teleoperation with time delay for an unmanned aerial vehicle involved in an obstacle avoidance task. In: AIAA modeling and simulation technologies conference

  32. Schill F, Hou X, Mahony R (2010) Admittance mode framework for haptic teleoperation of hovering vehicles with unlimited workspace. In: 2010 Australian conference on robotics and automation(ACRA)

  33. Hou X, Mahony R, Schill F (2010) Representation of vehicle dynamics in haptic teleoperation of aerial robots. In: 2013 IEEE international conference on robotics and automation(ICRA), IEEE, pp 1447–1483

  34. Cappe O, Moulines E, Ryden T (2006) Inference in hidden Markov models. Springer Science & Business Media, Berlin

    MATH  Google Scholar 

  35. Elliott RJ, Aggoun L, Moore JB (2008) Hidden Markov models: estimation and control, vol 29. Springer Science & Business Media, Berlin

    MATH  Google Scholar 

  36. Viterbi AJ (1967) Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Trans Inf Theory 13(2):260–269

    Article  Google Scholar 

  37. Rabiner L (1989) A tutorial on hidden markov models and selected applications in speech recognition. Proc IEEE 77(2):257–286

    Article  Google Scholar 

  38. LaValle SM, Kuffner JJ (1999) Randomized kinodynamic planning. In: 1999 IEEE international conference on robotics and automation (ICRA), IEEE, vol 1, pp 473–479

  39. Karaman S, Frazzoli E (2011) Sampling-based algorithms for optimal motion planning. Int J Robot Res 30(7):846–894

    Article  Google Scholar 

  40. Mahony R, Hamel T, Pflimlin JM (2008) Nonlinear complementary filters on the special orthogonal group. IEEE Trans Autom Control 53(5):1203–1218

    Article  MathSciNet  Google Scholar 

  41. Mahony R, Kumar V, Corke P (2012) Multirotor aerial vehicles: Modeling, estimation, and control of quadrotor. IEEE Robot Autom Mag 19(3):20–32

    Article  Google Scholar 

  42. Hamel T, Mahony R, Lozano R, Ostrowski J (2002) Dynamic modelling and configuration stabilization for an x4-flyer. IFAC World Congress 15(1):217–222

    Google Scholar 

  43. Pounds P, Mahony R, Corke P (2010) Modelling and control of a large quadrotor robot. Control Eng Pract 18(7):691–699

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Grant Nos. 61703343), Natural Science Foundation of Shaanxi Province (Grant Nos. 2018JQ6070) and Northwestern Polytechnical University “the Fundamental Research Funds for the Central Universities” (Grant Nos. 3102018JCC003).

Author information

Authors and Affiliations

  1. School of Automation, Northwestern Polytechnical University, Xi’an, China

    Xiaolei Hou

Authors
  1. Xiaolei Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaolei Hou.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hou, X. Haptic teleoperation of a multirotor aerial robot using path planning with human intention estimation. Intel Serv Robotics 14, 33–46 (2021). https://doi.org/10.1007/s11370-020-00339-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11370-020-00339-2

Keywords

Search

Navigation