A Visual-Haptic Display for Human and Autonomous Systems Integration

  • Matteo RazzanelliEmail author
  • Stefano Aringhieri
  • Giovanni Franzini
  • Giulio Avanzini
  • Fabrizio Giulietti
  • Mario Innocenti
  • Lorenzo Pollini
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9991)


This paper introduces a novel concept of visual-haptic display for situational awareness improvement for crowded and low altitude airspace situations. The visual augmentation display that constitutes of Virtual Fences delimiting no-fly zones, and a specific tri-dimensional highlight graphics that enhances visibility of other remotely piloted or autonomous agents, as well as conventional manned aircraft operating in the area is presented first. Then the Shared Control paradigm and the Haptic Force generation mechanism, based on a Proportional-Derivative-like controller applied to repulsive forces generated by the Virtual Fences and other UAVs are introduced and discussed. Simulations with 26 pilots were performed in a photo-realistic synthetic environment showing that the combined use of Visual-haptic feedback outperforms the Visual Display only in helping the pilot keeping a safe distance from no-fly zones and other vehicles.


Autonomous systems Human and autonomous systems teaming application Cooperative systems 


  1. 1.
    Sachs, G.: Perspective predictor/flight-path display with minimum pilot compensation. J. Guid. Control Dyn. 23(3), 420–429 (2000)CrossRefGoogle Scholar
  2. 2.
    Mulder, M., Veldhuijzen, A.R., van Paassen, M.M., Mulder, J.A.: Integrating fly-by-wire controls with perspective flight-path displays. J. Guid. Control Dyn. 28(6), 1263–1274 (2005)CrossRefGoogle Scholar
  3. 3.
    Giulietti, F., Pollini, L., Avanzini, A.: Visual aids for safe operation of remotely piloted vehicles in the controlled air space. In: Proceedings of Institution of Mechanical Engineers, Part G: Journal of Aerospace Engineering, Published online before print 7 March 2016 (2016). doi: 10.1177/0954410016632014
  4. 4.
    Innocenti, M., Pollini, L., Giulietti, F.: Visual tools for man-machine interface real time simulation. In: Proceedings of IEEE Symposium on Information Technology in Mechatronics, ITM 2001, Istanbul, Turkey, October 2001 (2001)Google Scholar
  5. 5.
    Avanzini, G.: Frenet-based algorithm for trajectory prediction. J. Guid. Control Dyn. 27(1), 127–135 (2004)CrossRefGoogle Scholar
  6. 6.
    Lam, T.M., Boschloo, H.W., Mulder, M., van Paassen, M.M.: Artificial force field for haptic feedback in UAV teleoperation. IEEE Trans. Syst. Man Cybern. Part A: Syst. Hum. 39(6), 1316–1330 (2009)CrossRefGoogle Scholar
  7. 7.
    Lam, T.M., Mulder, M., van Paassen, M.M., Mulder, J.A., van Der Helm, F.C.T., Force-stiffness feedback in UAV tele-operation with time delay. In: AIAA Guidance, Navigation, and Control Conference, Chicago, Illinois, August 2009Google Scholar
  8. 8.
    Diolaiti, N., Melchiorri, C.: Tele-operation of a mobile robot through haptic feedback. In: IEEE International Workshop on Haptic Virtual Environments and Their Applications (HAVE 2002), Ottawa, Ontario, Canada, November (2002)Google Scholar
  9. 9.
    Barnes, D.P., Counsell, M.S.: Haptic communication for remote mobile manipulator robot operations. In: Proceedings of 8th Topical Meeting on Robotics and Remote Systems. American Nuclear Society, Pittsburgh (1999)Google Scholar
  10. 10.
    Farkhatdinov, I., Ryu, J,-H., An, J.: A preliminary experimental study on haptic teleoperation of mobile robot with variable force feedback gain. In: Haptics Symposium, 25–26 March 2010. IEEE (2010)Google Scholar
  11. 11.
    Horan, B., Crelghton, D., Nahavandi, S., Jamshidi, M., Bilateral haptic teleoperation of an articulated track mobile robot. In: IEEE International Conference on System of Systems Engineering, SoSE 2007, 16–18 April 2007 (2007)Google Scholar
  12. 12.
    Mitsou, N.C., Velanas, S.V., Tzafestas, C.S.: Visuo-haptic interface for teleoperation of mobile robot exploration tasks. In: The 15th IEEE International Symposium on Robot and Human Interactive Communication, ROMAN 2006, 6–8 September 2006, pp. 157–163 (2006)Google Scholar
  13. 13.
    Rsch, O.J., Schilling, K., Roth, H.: Haptic interfaces for the remote control of mobile robots. Control Eng. Pract. 10(11), 1309–1313 (2002)CrossRefGoogle Scholar
  14. 14.
    Lee, S., Sukhatme, G.S., Kim, G.J., Park, C.-M.: Haptic control of a mobile robot: a user study. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 3, pp. 2867–2874 (2002)Google Scholar
  15. 15.
    De Stigter, S., Mulder, M., van Paassen, M.M.: Design and evaluation of a haptic flight director. J. Guid. Control Dyn. 30(1), 35–46 (2007)CrossRefGoogle Scholar
  16. 16.
    Abbink, D.A., Boer, E.R., Mulder, M.: Motivation for continuous haptic gas pedal feedback to support car following. In: Proceedings of IEEE Intelligent Vehicles Symposium, pp. 283–290 (2008)Google Scholar
  17. 17.
    Abbink, D., Mulder, M., Boer, E.R.: Haptic shared control: smoothly shifting control authority? Cogn. Technol. Work 14(1), 19–28 (2012)CrossRefGoogle Scholar
  18. 18.
    Goodrich, K., Schutte, P., Williams, R.: Haptic-multimodal flight control system update. In: Proceedings of 11th AIAA Aviation Technology, Integration, and Operations (ATIO) Conference, pp. 1–17 (2011)Google Scholar
  19. 19.
    Olivari, M., Nieuwenhuizen, F.M., Buelthoff, H.H., Pollini, L.: Pilot adaptation to different classes of haptic aids in tracking tasks. J. Guid. Control Dyn. 37(6), 1741–1753 (2014)CrossRefGoogle Scholar
  20. 20.
    Maimeri, M., Olivari, M., Buelthoff, H.H., Pollini, L.: On effects of failures in haptic and automated pilot support systems. In: Proceedings of AIAA Modeling and Simulation Technologies Conference, pp. 1–12 (2016)Google Scholar
  21. 21.
    D’Intino, G., Olivari, M., Geluardi, S., Venrooij, J., Innocenti, M., Buelthoff, H.H., Pollini, L.: Evaluation of haptic support system for training purposes in a tracking task. In: IEEE Systems, Men, and Cybernetics Society Conference (2016)Google Scholar
  22. 22.
    Petermeijer, S., Abbink, D., Mulder, M., de Winter, J.: The effect of haptic support systems on driver performance: a literature survey. IEEE Trans. Haptics 8(4), 467–479 (2015)CrossRefGoogle Scholar
  23. 23.
    Alaimo, S.M.C., Pollini, L., Bresciani, J.P., Blthoff, H.H.: A comparison of direct and indirect haptic aiding for remotely piloted vehicles. In: Proceedings of 19th IEEE International Symposium in Robot and Human Interactive Communication (IEEE Ro-Man 2010)Google Scholar
  24. 24.
    Alaimo, S.M.C., Pollini, L., Bresciani, J.-P., Blthoff, H.H.: Experiments of direct and indirect haptic aiding for remotely piloted vehicles with a mixed wind gust rejection/obstacle avoidance task. In: AIAA Modeling and Simulation Technologies Conference 2011 (MST-2011), Portland, Oregon (2011)Google Scholar
  25. 25.
    Profumo, L., Pollini, L., Abbink, D.A.: Direct and indirect haptic aiding for curve negotiation. In: IEEE International Conference on Systems, Man, and Cybernetics (2013)Google Scholar
  26. 26.
    Alaimo, S.M.C., Pollini, L., Innocenti, M., Bresciani, J.P., Buelthoff, H.H.: Experimental comparison of direct and indirect haptic aids in support of obstacle avoidance for remotely piloted vehicles. J. Mech. Eng. Autom. 2(10), 2159–5275 (2012)Google Scholar
  27. 27.
    Di Corato, F., Innocenti, M., Pollini, L.: Combined vision-inertial navigation for improved robustness. In: IEEE Israel Itzhack Y. Bar-Itzhack Memorial Symposium on Estimation, Navigation, and Spacecraft Control, Haifa, Israel, 14–17 October 2012Google Scholar
  28. 28.
    Di Corato, F., Innocenti, M., Pollini, L.: Robust vision-aided inertial navigation algorithm via entropy-like relative pose estimation. Gyroscopy Navig. 4(1), 1–13 (2013)CrossRefGoogle Scholar
  29. 29.
    Pollini, L., Metrangolo, A.: Simulation and robust backstepping control of a quadrotor aircraft. In: AIAA Modeling and Simulation Technologies Conference, Honolulu, Hawaii (2008)Google Scholar
  30. 30.
    Pollini, L., Innocenti, M.: A synthetic environment for dynamic systems control and distributed simulation. IEEE Control Syst. Mag. 20(2), 49–61 (2000)CrossRefGoogle Scholar
  31. 31.
    Hutton, J., Mostafa, M.M.R.: 10 years of direct georeferencing for airborne photogrammetry. GIS Bus. (GeoBit) 11(1), 33–41 (2005)Google Scholar
  32. 32.
    Nebiker, S., Eugster, H., Flckiger, K., Christen, M.: Planning and management of real-time geospatial UAS missions within a virtual globe environment. In: UAV-g 2011, Conference on Unmanned Aerial Vehicle in Geomatics (2011)Google Scholar
  33. 33.
    Pollini, L., Razzanelli, M., Olivari, M., Brandimarti, A., Maimeri, M., Pazzaglia, P., Pittiglio, G., Nuti, R., Innocenti, M., Buelthoff, H.H.: Design, realization and experimental evaluation of a haptic stick for shared control studies. In: The 13th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Human-Machine Systems, Kyoto, Japan (2016)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Matteo Razzanelli
    • 1
    Email author
  • Stefano Aringhieri
    • 1
  • Giovanni Franzini
    • 1
  • Giulio Avanzini
    • 2
  • Fabrizio Giulietti
    • 3
  • Mario Innocenti
    • 1
  • Lorenzo Pollini
    • 1
  1. 1.Dipartimento di Ingegneria dell’InformazioneUniversity of PisaPisaItaly
  2. 2.Dipartimento di Ingegneria dell’InnovazioneUniversity of SalentoLecceItaly
  3. 3.Dipartimento di Ingegneria IndustrialeUniversity of BolognaForlìItaly

Personalised recommendations