A Natural Interaction Interface for UAVs Using Intuitive Gesture Recognition

  • Meghan ChandaranaEmail author
  • Anna Trujillo
  • Kenji Shimada
  • B. Danette Allen
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 499)


The popularity of unmanned aerial vehicles (UAVs) is increasing as technological advancements boost their favorability for a broad range of applications. One application is science data collection. In fields like earth and atmospheric science, researchers are seeking to use UAVs to augment their current portfolio of platforms and increase their accessibility to geographic areas of interest. By increasing the number of data collection platforms, UAVs will significantly improve system robustness and allow for more sophisticated studies. Scientists would like the ability to deploy an available fleet of UAVs to traverse a desired flight path and collect sensor data without needing to understand the complex low-level controls required to describe and coordinate such a mission. A natural interaction interface for a Ground Control System (GCS) using gesture recognition is developed to allow non-expert users (e.g., scientists) to define a complex flight path for a UAV using intuitive hand gesture inputs from the constructed gesture library. The GCS calculates the combined trajectory on-line, verifies the trajectory with the user, and sends it to the UAV controller to be flown.


Natural interaction Gesture Trajectory Flight path UAV Non-expert user 



The authors would like to thank Javier Puig-Navarro and Syed Mehdi (NASA LaRC interns, University of Illinois at Urbana-Champaign) for their help in building the trajectory module, Gil Montague (NASA LaRC intern, Baldwin Wallace University) and Ben Kelley (NASA LaRC) for their expertise in integrating the DDS middleware, Dr. Loc Tran (NASA LaRC) for his insight on feature extraction, and the entire NASA LaRC Autonomy Incubator team for their invaluable support and feedback.


  1. 1.
    Jenkins, D., Bijan, B.: The economic impact of unmanned aircraft systems integration in the United States. Technical Report, AUVSI (2013)Google Scholar
  2. 2.
    Wald, M.L.: Domestic drones stir imaginations and concerns. New York Times (2013)Google Scholar
  3. 3.
    Barr, A.: Amazon testing delivery by drone, CEO Bezos Says. USA Today.
  4. 4.
    Saggiani, G., Teodorani, B.: Rotary wing UAV potential applications: an analytical study through a matrix method. Aircraft Eng. Aerosp. Technol. 76(1), 6–14 (2004)CrossRefGoogle Scholar
  5. 5.
    Office of the Secretary of Defense.: Unmanned aircraft systems roadmap 2005–2030. Washington, DC (2005)Google Scholar
  6. 6.
    Schoenwald, D.A.: AUVs: in space, air, water, and on the ground. IEEE Control Syst. Mag. 20(6), 15–18 (2000)Google Scholar
  7. 7.
    Chen, H., Wang, X., Li, Y.: A Survey of autonomous control for UAV. In: IEEE International Conference on Artificial Intelligence and Computational Intelligence, pp. 267–271. Shanghai (2009)Google Scholar
  8. 8.
    Hsu, J.: MIT researcher develops iPhone app to easily control swarms of aerial drones. In: Popular Science.
  9. 9.
    Zelinski, S., Koo, T.J., Sastry, S.: Hybrid system design for formations of autonomous vehicles. In: IEEE Conference on Decision and Control (2003)Google Scholar
  10. 10.
    Miniature Air Quality Monitoring System,
  11. 11.
    Wegener, S., Schoenung, et al.: UAV autonomous operations for airborne science missions. In: AIAA 3rd Unmanned Unlimited Technical Conference, Chicago (2004)Google Scholar
  12. 12.
    Wegener, S., Schoenung, S.: Lessons learned from NASA UAV science demonstration program missions. In: AIAA 2nd Unmanned Unlimited Systems, Technologies and Operations Conference, San Diego (2003)Google Scholar
  13. 13.
    Perzanowski, D., Schultz, A.C., et al.: Building a multimodal human-robot interaction. In: IEEE Intelligent Systems, pp. 16–21 (2001)Google Scholar
  14. 14.
    Trujillo, A.C., Fan, H., et al.: Operator informational needs for multiple autonomous small vehicles. In: 6th International Conference on Applied Human Factors and Ergonomics and the Affiliated Conferences, Las Vegas, pp. 5556–5563 (2015)Google Scholar
  15. 15.
    Trujillo, A.C., Cross, C., et al.: Collaborating with autonomous agents. In: 15th AIAA Aviation Technology, Integration and Operations Conference, Dallas (2015)Google Scholar
  16. 16.
    Reitsema, J., Chun, W., et al.: Team-centered virtual interaction presence for adjustable autonomy. In: AIAA Space, Long Beach (2005)Google Scholar
  17. 17.
    Wachs, J. P., K¨olsch, M., Stern, H., Edan, Y.: Vision-based hand-gesture applications. Commun. ACM 54(2), 60–71 (2011)Google Scholar
  18. 18.
    McCafferty, S.: Space for cognition: gesture and second language learning. Int. J. Appl. Linguist. 14(1), 148–165 (2004)CrossRefGoogle Scholar
  19. 19.
    Torrance, M.: Natural communication with robots. M.S. Thesis, Department of Electrical Engineering and Computer Science, MIT, Cambridge (1994)Google Scholar
  20. 20.
    Iba, S., Weghe, M.V., et al.: An architecture for gesture based control of mobile robots. In: IEEE International Conference on Intelligent Robots and Systems, pp. 851–857 (1999)Google Scholar
  21. 21.
    Neto, P., Pires, J.N.: High-level programming for industrial robotics: using gestures, speech and force control. In: IEEE International Conference on Robotics and Automation, Kobe (2009)Google Scholar
  22. 22.
    Yeo, Z.: GestureBots: intuitive control for small robots. In: CHI 28th Annual ACM Conference on Human Factors in Computing Systems, Atlanta (2010)Google Scholar
  23. 23.
    Waldherr, S., Romero, R., Thrun, S.: A gesture based interface for human-robot interaction. Auton. Robots 9, 151–173 (2000)CrossRefGoogle Scholar
  24. 24.
    Becker, M., Kefalea, E., et al.: GripSee: a gesture-controlled robot for object perception and manipulation. Auton. Robots 6, 203–221 (1999)Google Scholar
  25. 25.
    Lambrecht, J., Kleinsorge, M., Kruger, J.: Markerless gesture-based motion control and programming of industrial robots. In: 16th IEEE International Conference on Emerging Technologies and Factory Automation, Toulouse (2011)Google Scholar
  26. 26.
    Raheja, J.L., et al.: Real-time robotic hand control using hand gestures. In: IEEE 2nd International Conference on Machine Learning and Computing, Washington, DC (2010)Google Scholar
  27. 27.
    Bassily, D., et al.: Intuitive and adaptive robotic arm manipulation using the leap motion controller. In: 8th German Conference on Robotics, Munich (2014)Google Scholar
  28. 28.
    King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)Google Scholar
  29. 29.
    Choe, R., et al.: Trajectory generation using spatial pythagorean hodograph Bézier curves. In: AIAA Guidance, Navigation and Control Conference, Kissimmee (2015)Google Scholar
  30. 30.

Copyright information

© Springer International Publishing Switzerland 2017

Authors and Affiliations

  • Meghan Chandarana
    • 1
    Email author
  • Anna Trujillo
    • 2
  • Kenji Shimada
    • 1
  • B. Danette Allen
    • 2
  1. 1.Carnegie Mellon UniversityPittsburghUSA
  2. 2.NASA Langley Research CenterHamptonUSA

Personalised recommendations