Fingerspelling Recognition with Support Vector Machines and Hidden Conditional Random Fields

A Comparison with Neural Networks and Hidden Markov Models
  • César Roberto de Souza
  • Ednaldo Brigante Pizzolato
  • Mauro dos Santos Anjo
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7637)


In this paper, we describe our experiments with Hidden Conditional Random Fields and Support Vector Machines in the problem of fingerspelling recognition of the Brazilian Sign Language (LIBRAS). We also provide a comparison against more common approaches based on Artificial Neural Networks and Hidden Markov Models, reporting statistically significant results in k-fold cross-validation. We also explore specific behaviors of the Gaussian kernel affecting performance and sparseness. To perform multi-class classification with SVMs, we use large-margin Directed Acyclic Graphs, achieving faster evaluation rates. Both ANNs and HCRFs have been trained using the Resilient Backpropagation algorithm. In this work, we validate our results using Cohen’s Kappa tests for contingency tables.


gesture recognition fingerspelling sign languages LIBRAS support vector machines hidden conditional random fields neural networks hidden Markov models discriminative models 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Pizzolato, E.B., Anjo, M., Pedroso, G.: Automatic Recognition of Finger Spelling for LIBRAS Based on a Two-Layer Architecture. In: ACM Symposium on Applied Computing (SAC 2010), pp. 969–973. ACM (2010)Google Scholar
  2. 2.
    Platt, J., Cristianini, N., Shawe-Taylor, J.: Large Margin DAGs for Multiclass Classification. Advances in Neural Information Processing Systems 12(3), 547–553 (2000)Google Scholar
  3. 3.
    Starner, T.: Visual Recognition of American Sign Language Using Hidden Markov Models. Thesis (M.S.), Massachusetts Institute of Technology (1995)Google Scholar
  4. 4.
    Starner, T., Weaver, J., Pentland, A.: Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(12), 1371–1375 (1998)CrossRefGoogle Scholar
  5. 5.
    Bauer, B., Kraiss, K.-F.: Video-Based Sign Recognition Using Self-Organizing Subunits. In: 16th Intl. Conf. on Pattern Recognition (ICPR 2002), vol. 2, pp. 434–437. IEEE (2002)Google Scholar
  6. 6.
    Bowden, R., Windridge, D., Kadir, T., Zisserman, A., Brady, M.: A Linguistic Feature Vector for the Visual Interpretation of Sign Language. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3021, pp. 390–401. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  7. 7.
    Dias, D., Madeo, R., Rocha, T., Bíscaro, H., Peres, S.: Hand Movement Recognition for Brazilian Sign Language – A Study Using Distance-Based Neural Networks. In: Int. Joint Conf. on Neural Networks (IJCNN 2009), pp. 2355–2362. IEEE (2009)Google Scholar
  8. 8.
    Wang, S., Quattoni, A., Morency, L.-P., Demirdjian, D.: Hidden Conditional Random Fields for Gesture Recognition. In: 2006 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2006), pp. 1521–1527. IEEE (2006)Google Scholar
  9. 9.
    Morency, L.-P., Quattoni, A., Darrell, T.: Latent-Dynamic Discriminative Models for Continuous Gesture Recognition. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2007), pp. 1–8. IEEE (2007)Google Scholar
  10. 10.
    Elmezain, M., Al-Hamadi, A., Michaelis, B.: Discriminative Models-Based Hand Gesture Recognition. In: 2nd Intl. Conf. on Machine Vision (ICMV 2009), pp. 123–127. IEEE (2009)Google Scholar
  11. 11.
    Mitra, S., Acharya, T.: Gesture recognition: A survey. IEEE Trans. on Systems, Man and Cybernetics - Part C: Applications and Reviews 37(3), 311–324 (2007)CrossRefGoogle Scholar
  12. 12.
    Yang, H.-D., Sclaroff, S., Lee, S.-W.: Sign Language Spotting with a Threshold Model Based on Conditional Random Fields. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(7), 1264–1277 (2009)CrossRefGoogle Scholar
  13. 13.
    Juang, B., Rabiner, L.: Automatic Speech Recognition – A Brief History of the Technology. In: Encyclopedia of Language and Linguistics, 2nd edn., Elsevier (2005)Google Scholar
  14. 14.
    Ng, A., Jordan, M.: On Discriminative vs. Generative classifiers: A Comparison of Logistic Regression and Naive Bayes. In: Dietterich, T., Becker, S., Ghahramani, Z. (eds.) Neural Information Processing Systems, vol. 2, pp. 841–848. MIT Press (2001)Google Scholar
  15. 15.
    Quattoni, A., Collins, M., Darrell, T.: Conditional Random Fields for Object Recognition. In: Saul, L., Weiss, Y., Bottou, L. (eds.) Advances in Neural Information Processing Systems, vol. 17, pp. 1097–1104 (2005)Google Scholar
  16. 16.
    Sutton, C., McCallum, A.: Introduction to Statistical Relational Learning. In: Taskar, L. (ed.) An Introduction to Conditional Random Fields for Relational Learning. MIT Press (2007)Google Scholar
  17. 17.
    Igel, C., Hüsken, M.: Improving the Rprop Learning Algorithm. In: 2nd Intl. ICSC Symposium on Neural Computation (NC 2000), pp. 115–121 (2000)Google Scholar
  18. 18.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. Springer (2009)Google Scholar
  19. 19.
    Huang, L.: Variable Selection in Multi-Class Support Vector Machine and Applications in Genomic Data Analysis. ProQuest (2008)Google Scholar
  20. 20.
    Michel, P., El Kaliouby, R.: Real Time Facial Expression Recognition in Video Using Support Vector Machines. In: 5th International Conf. on Multimodal Interfaces (ICMI 2003), pp. 258–264. ACM (2003)Google Scholar
  21. 21.
    Cortes, C., Vapnik, V.: Support-Vector Networks. Machine Learning 20(3), 273–297 (1995)MATHGoogle Scholar
  22. 22.
    Rabiner, L.R.: A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. In: Waibel, A., Lee, K.-F. (eds.) Readings in Speech Recognition, pp. 267–296. Morgan Kaufmann Publishers Inc. (1990)Google Scholar
  23. 23.
    Lafferty, J., McCallum, A., Pereira, F.: Conditional Random Fields – Probabilistic Models for Segmenting and Labeling Sequence Data. In: 18th Intl. Conf. on Machine Learning (ICLM 2001), pp. 282–289. ACM (2001)Google Scholar
  24. 24.
    Platt, J.: Sequential Minimal Optimization – A Fast Algorithm for Training Support Vector Machines. Advances in Kernel Methods and Support Vector Learning 208, 1–21 (1998)Google Scholar
  25. 25.
    Caputo, B., Sim, K., Furesjo, F., Smola, A.: Appearance-Based Object Recognition Using SVMs – Which Kernel Should I Use? In: NIPS Workshop on Statistical Methods for Computational Experiments in Visual Processing and Computer Vision (2002)Google Scholar
  26. 26.
    Valentini, G., Dietterich, T.: Bias-Variance Analysis of Support Vector Machines for the Development of SVM-Based Ensemble Methods. The Journal of Machine Learning Research 5, 725–775 (2004)MathSciNetMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • César Roberto de Souza
    • 1
  • Ednaldo Brigante Pizzolato
    • 1
  • Mauro dos Santos Anjo
    • 1
  1. 1.Universidade Federal de São CarlosSão CarlosBrasil

Personalised recommendations