Current feature-based gesture recognition systems use human-chosen features to perform recognition. Effective features for classification can also be automatically learned and chosen by the computer. In other recognition domains, such as face recognition, manifold learning methods have been found to be good nonlinear feature extractors. Few manifold learning algorithms, however, have been applied to gesture recognition. Current manifold learning techniques focus only on spatial information, making them undesirable for use in the domain of gesture recognition where stroke timing data can provide helpful insight into the recognition of hand-drawn symbols. In this paper, we develop a new algorithm for multi-stroke gesture recognition, which integrates timing data into a manifold learning algorithm based on a kernel Isomap. Experimental results show it to perform better than traditional human-chosen feature-based systems.


Sketch Recognition Manifold Learning Kernel Isomap 


  1. 1.
    LaViola, J., Zeleznik, R.: Mathpad2: A system for the creation and exploration of mathematical sketches. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 23(3) (2004)Google Scholar
  2. 2.
    Stahovich, T., Davis, R., Shrobe, H.: Qualitative rigid body mechanics. Artificial Intelligence (2000)Google Scholar
  3. 3.
    Landay, J.A., Myers, B.A.: Sketching interfaces: Toward more human interface design. IEEE Computer 34(3), 56–64 (2001)CrossRefGoogle Scholar
  4. 4.
    Forbus, K.D., Usher, J., Chapman, V.: Sketching for military course of action diagrams. In: Proceedings of IUI 2003 (2003)Google Scholar
  5. 5.
    Do, E.Y.L.: VR sketchpad - create instant 3D worlds by sketching on a transparent window. In: de Vries, B., van Leeuwen, J.P., Achten, H.H. (eds.) CAAD Futures 2001, pp. 161–172 (July 2001)Google Scholar
  6. 6.
    Forsberg, A.S., Dieterich, M.K., Zeleznik, R.C.: The music notepad. In: Proceedings of UIST 1998, ACM SIGGRAPH (1998)Google Scholar
  7. 7.
    Igarashi, T., Matsuoka, S., Tanaka, H.: Teddy: A sketching interface for 3d freeform design. In: SIGGRAPH 1999, pp. 409–416 (August 1999)Google Scholar
  8. 8.
    Mahoney, J.V., Fromherz, M.P.J.: Interpreting sloppy stick figures by graph rectification and constraint-based matching. In: Fourth IAPR Int. Workshop on Graphics Recognition, Kingston, Ontario, Canada (2001)Google Scholar
  9. 9.
    Muzumdar, M.: ICEMENDR: Intelligent capture environment for mechanical engineering drawing. Master’s thesis, Massachusetts Institute of Technology (1999)Google Scholar
  10. 10.
    Hammond, T., Davis, R.: Tahuti: A geometrical sketch recognition system for UML class diagrams. In: AAAI Spring Symposium on Sketch Understanding, March 25-27, pp. 59–68 (2002)Google Scholar
  11. 11.
    Patel, R., Plimmer, B., Grundy, J., Ihaka, R.: Ink features for diagram recognition. In: Sketch Based Interfaces and Modeling IEEE, Eurographics (2007)Google Scholar
  12. 12.
    Rubine, D.: Specifying gestures by example. Computer Graphics 25(4), 329–337 (1991)CrossRefGoogle Scholar
  13. 13.
    Long, A.C., Landay, J.A., Rowe, L.A., Michiels, J.: Visual similarity of pen gestures. In: Human Factors in Computing Systems (2000)Google Scholar
  14. 14.
    Sezgin, T.M., Stahovich, T., Davis, R.: Sketch based interfaces: Early processing for sketch understanding. In: Proceedings of 2001 Perceptive User Interfaces Workshop (PUI 2001) (2001)Google Scholar
  15. 15.
    Mahoney, J.V., Fromherz, M.P.J.: Three main concerns in sketch recognition and an approach to addressing them. In: AAAI Spring Symposium on Sketch Understanding, Standord, CA, pp. 105–112 (March 2002)Google Scholar
  16. 16.
    Rabiner, L.R., Juang, B.H.: An introduction to hidden Markov models. IEEE Trans. Acoustics, Speech, and Signal Processing Magazine 3, 4–16 (1986)Google Scholar
  17. 17.
    Sezgin, T.M.: Sketch Interpretation Using Multiscale Stochastic Models of Temporal Patterns. PhD thesis, Massachusetts Institute of Technology (May 2006)Google Scholar
  18. 18.
    Sun, Z., Jiang, W., Sun, J.: Adaptive online multi-stroke sketch recognition based on hidden markov model. In: Yeung, D.S., Liu, Z.-Q., Wang, X.-Z., Yan, H. (eds.) ICMLC 2005. LNCS, vol. 3930, pp. 948–957. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  19. 19.
    Muller, S., Eickeler, S., Rigoll, G.: Image database retrieval of rotated objects by user sketch. In: IEEE Workshop on Content-Based Access of Image and Video Libraries, p. 40 (1998)Google Scholar
  20. 20.
    Alvarado, C., Davis, R.: Sketchread: A multi-domain sketch recognition engine. In: Proceedings of UIST 2004, pp. 23–32 (2004)Google Scholar
  21. 21.
    Hammond, T., Davis, R.: Ladder, a sketching language for user interface developers. Elsevier, Computers and Graphics 28, 518–532 (2005)CrossRefGoogle Scholar
  22. 22.
    Kara, L.B., Stahovich, T.F.: An image-based trainable symbol recognizer for sketch-based interfaces. In: Making Pen-Based Interaction Intelligent and Natural, Menlo Park, California, October 21-24. AAAI Fall Symposium, pp. 99–105 (2004)Google Scholar
  23. 23.
    Lee, W., Kara, L.B., Stahovich, T.F.: An efficient graph-based recognizer for hand-drawn symbols. Computers & Graphics 31, 554–567 (2007)CrossRefGoogle Scholar
  24. 24.
    Seung, H.S., Lee, D.D.: The manifold ways of perception. Science 290, 2268–2269 (2000)CrossRefGoogle Scholar
  25. 25.
    Tenenbaum, J.B., de Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290, 2319–2323 (2000)CrossRefGoogle Scholar
  26. 26.
    Saul, L., Roweis, S.T.: Think globally, fit locally: Unsupervised learning of low dimensional manifolds. Journal of Machine Learning Research 4, 119–155 (2003)MathSciNetzbMATHGoogle Scholar
  27. 27.
    Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation 15, 1373–1396 (2003)CrossRefzbMATHGoogle Scholar
  28. 28.
    de Silva, V., Tenenbaum, J.B.: Global versus local methods in nonlinear dimensionality reduction. In: Advances in Neural Information Processing Systems, vol. 15, pp. 705–712. MIT Press, Cambridge (2003)Google Scholar
  29. 29.
    Jenkins, O.C., Matari, M.J.: A spatio-temporal extension to isomap nonlinear dimension reduction. In: Proc. Int’l. Conf. Machine Learning, Banff, Canada (2004)Google Scholar
  30. 30.
    Wobbrock, J., Wilson, A., Li, Y.: Gestures without libraries, toolkits, or training: A $1 recognizer for user interface prototypes. In: Proc. of the 20th Annual ACM Symposium on User Interface Software and Technology, Newport, RI, USA (2007)Google Scholar
  31. 31.
    Choi, H., Choi, S.: Robust Kernel Isomap. Pattern Recognition 40(3), 853–862 (2007)CrossRefzbMATHGoogle Scholar
  32. 32.
    Girolaini, M.: Mercer kernel-based clustering in feature space. IEEE Transactions on Neural Networks 13(3), 780–784 (2002)CrossRefGoogle Scholar
  33. 33.
    Schölkopf, B., Smola, A.J., Müller, K.R.: Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation 10(5), 1299–1319 (1998)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Heeyoul Choi
    • 1
  • Brandon Paulson
    • 1
  • Tracy Hammond
    • 1
  1. 1.Dept. of Computer ScienceTexas A&M UniversityUSA

Personalised recommendations