Multi-Touch Gesture Recognition Using Feature Extraction

  • Francisco R. Ortega
  • Naphtali Rishe
  • Armando Barreto
  • Fatemeh Abyarjoo
  • Malek Adjouadi
Conference paper
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 313)


We are motivated to find a multi-touch gesture detection algorithm that is efficient, easy to implement, and scalable to real-time applications using 3D environments. Our approach tries to solve the recognition for gestures with the use of feature extraction without the need of any previous learning samples. Before showing our proposed solution, we describe some algorithms that attempt to solve similar problems. Finally, we describe our code to accomplish off-line gesture recognition.


Multi-touch Feature extractions User interfaces 3D user interfaces Human-computer interaction Multi-touch recognition 



This work was sponsored by NSF grants HRD-0833093, and CNS-0959985. Mr. Francisco Ortega is the recipient of a GAANN fellowship, from the US Department of Education, at Florida International University.


  1. 1.
    J. Wobbrock and A. Wilson, “Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes,” Proceedings of the 20th annual ACM symposium on User interface software and technology (UIST ’07), 2007.Google Scholar
  2. 2.
    D. Rubine, “Specifying gestures by example,” ACM SIGGRAPH Computer Graphics, vol. 25, no. 4, pp. 329–337, 1991.CrossRefGoogle Scholar
  3. 3.
    G. Nielson and D. Olsen Jr, “Direct manipulation techniques for 3D objects using 2D locator devices,” Proceedings of the 1986 workshop on Interactive 3D graphics, pp. 175–182, 1987.Google Scholar
  4. 4.
    M. Chen, S. J. Mountford, and A. Sellen, “A study in interactive 3-d rotation using 2-d control devices,” in Proceedings of the 15th Annual Conference on Computer Graphics and Interactive Techniques, ser. SIGGRAPH ’88. New York, NY, USA: ACM, 1988, pp. 121–129.Google Scholar
  5. 5.
    H. Lü and Y. Li, “Gesture avatar: a technique for operating mobile user interfaces using gestures,” in CHI ’11: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM Request Permissions, May 2011.Google Scholar
  6. 6.
    P. Hong and T. Huang, “Constructing finite state machines for fast gesture recognition,” 15th International Conference on Pattern Recognition (ICPR’00), vol. 3, p. 3695, 2000.Google Scholar
  7. 7.
    P. Hong, T. Huang, and M. Turk, “Gesture modeling and recognition using finite state machines,” IEEE Conference on Face and Gesture Recognition, Mar. 2000.Google Scholar
  8. 8.
    T. Sezgin and R. Davis, “HMM-based efficient sketch recognition,” Proceedings of the 10th international conference on Intelligent user interfaces (IUI ’05), 2005.Google Scholar
  9. 9.
    J. Pittman, “Recognizing handwritten text,” in Human factors in computing systems: Reaching through technology (CHI ’91), New York, NY, 1991, pp. 271–275.Google Scholar
  10. 10.
    S. MacLean and G. Labahn, “Elastic matching in linear time and constant space,” International Workshop on Document Analysis Systems 2010 (DAS ’10), 2010.Google Scholar
  11. 11.
    L. Kara and T. Stahovich, “An image-based, trainable symbol recognizer for hand-drawn sketches,” Computers & Graphics, vol. 29, no. 4, pp. 501–517, 2005.CrossRefGoogle Scholar
  12. 12.
    L. Anthony and J. Wobbrock, “A lightweight multistroke recognizer for user interface prototypes,” in Proceedings of Graphics Interface 2010 (GI’10), Toronto, ON, 2010.Google Scholar
  13. 13.
    Y. Li, “Protractor: a fast and accurate gesture recognizer,” in Proceedings of the 28th international conference on Human factors in computing systems (CHI ’10), New York, NY, 2010.Google Scholar
  14. 14.
    G. Johnson, M. Gross, and J. Hong, “Computational support for sketching in design: a review,” Foundations and Trends in Human-Computer Interaction 2, 2009.Google Scholar
  15. 15.
    C. C. Tappert, C. Y. Suen, and T. Wakahara, “The state of the art in online handwriting recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 12, no. 8, pp. 787–808, Aug. 1990.Google Scholar
  16. 16.
    R. Plamondon and S. N. Srihari, “Online and off-line handwriting recognition: a comprehensive survey,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 22, no. 1, pp. 63–84, 2000.Google Scholar
  17. 17.
    M. Blumenstein, B. Verma, and H. Basli, “A novel feature extraction technique for the recognition of segmented handwritten characters,” in Document Analysis and Recognition, 2003. Proceedings. Seventh International Conference on, 2003, pp. 137–141.Google Scholar
  18. 18.
    B. Signer, U. Kurmann, and M. C. Norrie, “iGesture: A General Gesture Recognition Framework,” in Document Analysis and Recognition, 2007. ICDAR 2007. Ninth International Conference on, 2007, pp. 954–958.Google Scholar
  19. 19.
    S. Kratz and M. Rohs, “Protractor3d: A closed-form solution to rotation-invariant 3d gestures,” in Proceedings of the 16th International Conference on Intelligent User Interfaces, ser. IUI ’11. New York, NY, USA: ACM, 2011, pp. 371–374.Google Scholar
  20. 20.
    S. Kratz and M. Rohs. “The $3 recognizer: Simple 3d gesture recognition on mobile devices,” in Proceedings of the 15th International Conference on Intelligent User Interfaces, ser. IUI ’10. New York, NY, USA: ACM, 2010, pp. 419–420.Google Scholar
  21. 21.
    W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes 3rd Edition: The Art of Scientific Computing, 3rd ed. New York, NY, USA: Cambridge University Press, 2007.Google Scholar
  22. 22.
    F. Wang, X. Cao, X. Ren, and P. Irani, “Detecting and leveraging finger orientation for interaction with direct-touch surfaces,” Proceedings of the 22nd annual ACM symposium on User interface software and technology, pp. 23–32, 2009.Google Scholar
  23. 23.
    F. R. Ortega, A. Barreto, N. Rishe, and M. Adjouadi, “To- wards 3D Data Environments using Multi-Touch Screens,” in ACHI 2012: The Fifth International Conference on Advances in Computer-Human Interactions, 2012, pp. 118–121.Google Scholar
  24. 24.
    (2012, 11). [Online]. Available:
  25. 25.
    Y. Kiriaty, L. Moroney, S. Goldshtein, and A. Fliess, Introducing Windows 7 for Developers. Microsoft Pr, Sep. 2009.Google Scholar
  26. 26.
    M. Sipser, Introduction to Theory of Computation, 2nd ed.Cengage, 2006.Google Scholar
  27. 27.
    A. Williams, C++ Concurrency in Action: Practical Multithreading, 1st ed. Manning Publications, Feb. 2012.Google Scholar
  28. 28.
    G. Gan, C. Ma, and J. Wu, Data Clustering: Theory, Algorithms, and Applications (ASA-SIAM Series on Statistics and Applied Probability). SIAM, Society for Industrial and Applied Mathematics, May 2007.Google Scholar
  29. 29.
    M. Herlihy and N. Shavit, The Art of Multiprocessor Programming, 1st ed. Morgan Kaufmann, Mar. 2008.Google Scholar
  30. 30.
    F. Dunn and I. Parberry, 3D Math Primer for Graphics and Game Development, 2nd Edition, 2nd ed. A K Peters/CRC Press, Nov. 2011.Google Scholar
  31. 31.
    W. Buxton, “A three-state model of graphical input,” Human-computer interaction-INTERACT, vol. 90, pp. 449–456, 1990.Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Francisco R. Ortega
    • 1
  • Naphtali Rishe
    • 1
  • Armando Barreto
    • 2
  • Fatemeh Abyarjoo
    • 2
  • Malek Adjouadi
    • 2
  1. 1.School of Computing and Information SciencesFlorida International UniversityMiamiUSA
  2. 2.Electrical and Computer Engineering DepartmentFlorida International UniversityMiamiUSA

Personalised recommendations