Journal on Multimodal User Interfaces

, Volume 12, Issue 1, pp 31–39 | Cite as

Ultrasonic hand gesture recognition for mobile devices

  • Mohamed Saad
  • Chris J. Bleakley
  • Vivek Nigram
  • Paul Kettle
Original Paper
  • 164 Downloads

Abstract

This paper describes a novel system for ultrasonic gesture recognition targeted at handheld devices, such as smartphones. Unlike previous proposals, the system obtains high update rate range estimates for the user’s hand. The range of the user’s hand is determined based on the Round Trip Time of ultrasonic pulses emitted by a transducer on the device, reflected by the hand and received at multiple sensors on the device. The range estimates, coupled with other information extracted from the reflected ultrasonic signals, are used for gesture recognition. Gesture recognition is performed by means of a multi-class hierarchical binary support vector machine. The high update rate is enabled by the use of compact wideband transducers. The ultrasonic pulses are short in duration and utilize Linear Frequency Modulation compression to achieve high resolution in Time Of Arrival estimation. The impact of multipath is reduced by the use of Frequency Hopping. A system prototype using one transmitter and four receivers was found to achieve gesture detection sensitivity and specificity of 99% and 99%, respectively, and classification accuracy of 96% for 7 users (5 males, 2 females) with around 500 repetitions per user for a set of 7 gesture types.

Keywords

Gesture-based interaction Ultrasonic Support vector machine (SVM) Human–Computer Interface 

References

  1. 1.
    Mitra S, Acharya T (2007) Gesture recognition: a survey. IEEE Trans Syst Man Cybern Part C Appl Rev 37(3):311–324CrossRefGoogle Scholar
  2. 2.
    Gonzalez J, Bleakley C (2011) Low cost, wideband ultrasonic transmitter and receiver for array signal processing applications. IEEE Sens J 11:1284–1292CrossRefGoogle Scholar
  3. 3.
    Murata open-structure ultrasonic broadband transmitter. In: http://www.murata.com/en-global/products/sensor. Murata website visited 03-11-2014
  4. 4.
    Surface mount MEMS: Model-ID SPU0410LR5H. In: http://www.knowles.com/eng/Products/Microphones/Surface-mount-MEMs. Knowles Acoustics website visited 03-11-2014
  5. 5.
    Biber C, Ellin S, Shenk E, Stempeck J (1980) The polaroid ultrasonic ranging system. In: Audio engineering society convention 67. Audio Engineering SocietyGoogle Scholar
  6. 6.
    Fléty E (2000) 3D gesture acquisition using ultrasonic sensors. Trends in Gestural Control of Music 193–207. Institute of Research and Acoustic Coordination Music-Center Pompidou, Paris, FranceGoogle Scholar
  7. 7.
    Kalgaonkar K, Raj B (2009) One-handed gesture recognition using ultrasonic doppler sonar. In: IEEE international conference on acoustics, speech and signal processing, 2009, ICASSP 2009. IEEE, pp 1889–1892Google Scholar
  8. 8.
    Kreczmer B (2011) Gestures recognition by using ultrasonic range-finders. In: 2011 16th International conference on methods and models in automation and robotics (MMAR). IEEE, pp 363–368Google Scholar
  9. 9.
    Gupta S, Morris D, Patel S, Tan D (2012) Soundwave: using the doppler effect to sense gestures. In: Proceedings of the 2012 ACM annual conference on human factors in computing systems. ACM, pp 1911–1914Google Scholar
  10. 10.
    Przybyla R, Shelton S, Guedes A, Krigel R, Horsley D, Boser B (2012) In-air ultrasonic rangefinding and angle estimation using an array of ALN micromachined transducers. In: Solid-state sensors, actuators and microsystems workshop, vol. 7, pp 3–7Google Scholar
  11. 11.
    Przybyla RJ, Tang H-Y, Shelton SE, Horsley DA, Boser BE (2014) 3d ultrasonic gesture recognition. In: 2014 IEEE international solid-state circuits conference digest of technical papers (ISSCC). IEEE, pp 210–211Google Scholar
  12. 12.
    Chiu T-I, Deng H-C, Chang S-Y, Luo S-B (2009) Implementation of ultrasonic touchless interactive panel using the polymer-based CMUT array. In: 2009 IEEE Sensors. IEEE, pp 625–630Google Scholar
  13. 13.
    Dahl T, Ealo JL, Pazos-Ospina J, Syversrud C, Kavli T, Bang HJ (2012) High-resolution ultrasonic gesture tracking systems for future portable devices. In: IEEE International ultrasonics symposium (IUS). pp. 150–153Google Scholar
  14. 14.
    Shelton S, Rozen O, Guedes A, Przybyla R, Boser B, Horsley D (2014) Improved acoustic coupling of air-coupled micromachined ultrasonic transducers. In: 2014 IEEE 27th International conference on micro electro mechanical systems (MEMS). pp 753–756Google Scholar
  15. 15.
    Ultrasound gesture recognition. In: http://www.ellipticlabs.com. Elliptic Labs website visited 26-01-2015
  16. 16.
    Grandke T (1983) Interpolation algorithms for discrete Fourier transforms of weighted signals. IEEE Trans Instrum Meas 32(2):350–355CrossRefGoogle Scholar
  17. 17.
    Gonzalez J, Bleakley C (2009) High-precision robust broadband ultrasonic location and orientation estimation. IEEE J Sel Top Signal Process 3(5):832–844CrossRefGoogle Scholar
  18. 18.
    Saad M, Bleakley C, Dobson S (2011) Robust high-accuracy ultrasonic range measurement system. IEEE Trans Instrum Meas 60(10):3334–3341CrossRefGoogle Scholar
  19. 19.
    LynxTWO 192 khz multichannel audio reference interface. In: http://www.lynxstudio.com/product_detail.asp?i=12. Lynx Studio Technology website visited 26-01-2015
  20. 20.
    iphone 7. In:https://browser.primatelabs.com/v4/cpu/387526. Geekbench Browser website visited 08-05-2017

Copyright information

© Springer International Publishing AG, part of Springer Nature 2017

Authors and Affiliations

  • Mohamed Saad
    • 1
  • Chris J. Bleakley
    • 1
  • Vivek Nigram
    • 2
  • Paul Kettle
    • 2
  1. 1.UCD School of Computer ScienceUniversity College DublinBelfield, Dublin 4Ireland
  2. 2.Maxim IntegratedSan JoseUSA

Personalised recommendations