Skip to main content
SpringerLink
Log in

A system for recognizing online handwritten mathematical expressions by using improved structural analysis

  • Original Paper
  • Published:
International Journal on Document Analysis and Recognition (IJDAR) Aims and scope Submit manuscript

Abstract

A system for recognizing online handwritten mathematical expressions (MEs), by applying improved structural analysis, is proposed and experimentally evaluated on two databases. With this system, MEs are represented in the form of stochastic context-free grammar (SCFG), and the Cocke–Younger–Kasami (CYK) algorithm is used to parse two-dimensional (2D) structures of online handwritten MEs and select the best interpretation in terms of the results of symbol segmentation and recognition as well as structural analysis. A concept of “body box” is proposed, and two SVM models are applied for learning and analyzing structural relations from training patterns without the need for any heuristic decisions. Stroke order is used to reduce the complexity of the parsing algorithm. Even though SCFG does not resolve ambiguities in some cases, the proposed system still gives users a list of candidates that contains the expected result. The results of experimental evaluations of the proposed system on the CROHME 2013 and CROHME 2014 databases and on an in-house (“Hand-Math”) database show that the recognition rate of the proposed system is improved, while the processing time on a common CPU is kept to a practical level.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Ruzzo, W.L.: On the complexity of general context-free language parsing and recognition, In: Automata, Languages, and Programming, pp. 489–497. Lecture Notes in Computer Science. Springer Verlag (1979)

  2. Anderson, R.H.: Syntax-directed recognition of handprinted two-dimensional mathematics. Ph.D. thesis, Harvard University (1968)

  3. Chang, S.K.: A method for the structural analysis of two-dimensional mathematical expressions. Int. J. Inf. Sci. 2, 253–272 (1970)

    MATH  Google Scholar 

  4. Chan, K.F., Yeung, D.Y.: Mathematical expression recognition: a survey. Int. J. Doc. Anal. Recogn. 3, 3–15 (2000)

    Article  Google Scholar 

  5. Zanibbi, R., Blostein, D.: Recognition and retrieval of mathematical expressions. Int. J. Doc. Anal. Recogn. 15, 331–357 (2012)

    Article  Google Scholar 

  6. Lehmberg, S., Winkler, H.J., Lang, M.: A soft-decision approach for symbol segmentation within handwritten mathematical expressions. In: Proceeding of international conference on acoustics, speech, and signal processing, vol. 6, pp. 3434–3437. Atlanta (1996)

  7. Toyozumi, K., et al.: A study of symbol segmentation method for handwritten mathematical formula recognition using mathematical structure information. In: Proceedings of the 17th ICPR, vol. 2, pp. 630–633. Cambridge (2004)

  8. Shi, Y., Li, H., Soong, F.K.: A unified framework for symbol segmentation and recognition of handwritten mathematical expressions. In: Proceedings of the 9th ICDAR, vol. 2, pp. 854–858. Curitiba (2007)

  9. Hu, L., Zanibbi, R.: Segmenting handwritten math symbols using adaboost and multi-scale shape context features. In: Proceedings of the 12th ICDAR, pp. 1180–1184. Washington (2013)

  10. MacLean, S., Labahn, G.: Elastic matching in linear time and constant space. In: Proceedings of the 9th IAPR Workshop on Document Analysis Systems. (2010)

  11. Hu, L., Zanibbi, R.: HMM-based recognition of online hand-written mathematical symbols using segmental K-means initialization and a modified pen-up/down feature. In: Proceedings of the 11th ICDAR, pp. 457–462. Beijing (2011)

  12. Álvaro, F., Sánchez, J. A., Benedí, J. M.: Classification of on-line mathematical symbols with hybrid features and recurrent neural networks. In: Proceedings of the 12th ICDAR, pp. 1012–1016. Washington (2013)

  13. Davila, K.M., Ludi, S., Zanibbi R.: Using off-line features and synthetic data for on-line handwritten math symbol recognition. In: Proceedings of the 14th ICFHR, pp. 323–328. Crete (2014)

  14. Garain, U., Chaudhuri, B.B.: Recognition of online handwritten mathematical expressions. IEEE Trans. Syst. Man Cybern. B Cybern. 34, 2366–2376 (2004)

    Article  Google Scholar 

  15. Álvaro, F., Sánchez, J.A., Benedí, J.M.: Offline features for classifying handwritten math symbols with recurrent neural networks. In: Proceedings of the 22nd ICPR, pp. 2944–2949. Stockholm (2014)

  16. Okamoto, M., Miao, B.: Recognition of mathematical expressions by using the layout structure of symbols. In: Proceedings of the 1st ICDAR, pp. 242–250. Saint Malo (1991)

  17. Eto, T., Suzuki, M.: Mathematical formula recognition using virtual link network. In: Proceedings of the 6th ICDAR, pp. 762–767. Seattle (2001)

  18. Aly, W., Uchida, S., Fujiyoshi, A., Suzuki, M.: Statistical classification of spatial relationships among mathematical symbols. In: Proceedings of the 10th ICDAR, pp. 1350–1354. Barcelona (2009)

  19. Álvaro, F., Zanibbi, R.: A shape-based layout descriptor for classifying spatial relationships in handwritten math. In: Proceedings of the ACM Symposium on Document Engineering. Florence (2013)

  20. Zanibbi, R., Blostein, D., Cordy, J.R.: Recognizing mathematical expressions using tree transformation. IEEE Trans. PAMI, 24, 1455–1467 (2002)

  21. Rhee, T.H., Kim, J.H.: Efficient search strategy in structural analysis for handwritten mathematical expression recognition. Pattern Recogn. 42, 3192–3201 (2009)

    Article  MATH  Google Scholar 

  22. Yamamoto, R., Sako, S., Nishimoto, T., Sagayama, S.: OnLine recognition of handwritten mathematical expressions based on stroke-based stochastic context-free grammar. In: 10th IWFHR, pp. 249–254. La Baule (2006)

  23. Simistira, F., Katsouros, V., Carayannis, G.: Recognition of online handwritten mathematical formulas using probabilistic SVMs and stochastic context free grammars. Pattern Recogn. Lett. 53, 85–92 (2015)

    Article  Google Scholar 

  24. MacLean, S., Labahn, G.: A New approach for recognizing handwritten mathematics using relational grammars and fuzzy sets. Int. J. Doc. Anal. Recogn. 16, 139–163 (2013)

    Article  Google Scholar 

  25. MacLean, S., Labahn, G.: A Bayesian model for recognizing handwritten mathematical expressions. Pattern Recogn. 48, 2433–2445 (2015)

    Article  Google Scholar 

  26. Álvaro, F., Sánchez, J.A., Benedí, J.M.: Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden Markov models. Pattern Recogn. Lett. 35, 58–67 (2014)

    Article  Google Scholar 

  27. Awal, A.M., Mouchére, H., Viard-Gaudin, C.: A global learning approach for an online handwritten mathematical expression recognition system. Pattern Recogn. Lett. 35, 6877 (2014)

    Article  Google Scholar 

  28. Mouchère, H., Viard-Gaudin, C., Zanibbi, R., Garain, U., Kim, D.H., Kim, J.H.: ICDAR 2013 CROHME: third international competition on recognition of online handwritten mathematical expressions. In: Proceedings of the 12th ICDAR, pp. 1428–1432. (2013)

  29. Mouchère, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: ICFHR: competition on recognition of on-line handwritten mathematical expressions (CROHME 2014). In: Proceedings of the 14th ICHFR, pp. 791–796. Crete (2014)

  30. Anthony, L., Yang, J., Koedinger, K.R.: Evaluation of multi modal input for entering mathematical equations on the computer. In: HI ’05 Extended Abstracts on Human Factors in Computing Systems, pp. 1184–1187

  31. Anthony, L., Yang, J., Koedinger, K.R.: A paradigm for handwriting-based intelligent tutors. Int. J. Hum. Comput. Stud. 70, 866–887 (2012)

    Article  Google Scholar 

  32. Le, A.D., Nakagawa, M.: A tool for ground-truthing online handwritten mathematical expressions. In: Proceedings of the 16th International Graphonomics Society Conference, pp. 70–73. (2013)

  33. Zhu, B., Gao, J., Nakagawa, M.: Objective function design for MCE-based combination of on-line and off-line character recognizers for on-line handwritten Japanese text recognition. In: Proceedings of the 11th ICDAR, pp. 594–599. (2011)

  34. Lari, K., Young, S.J.: The estimation of stochastic context-free grammars using the inside–outside algorithm. Comput. Speech Lang. 4, 35–56 (1990)

    Article  Google Scholar 

  35. Le, A.D., Phan, T.V., Nakagawa, M.: A System for recognizing online handwritten mathematical expressions and improvement of structure analysis. In: Proceedings of the 11th IAPR Workshop on Document Analysis Systems, pp. 51–55. (2014)

Download references

Acknowledgments

This research is being supported partially by Grant-in-Aid for Scientific Research under contract No. (B) 24300095, NEDO under the No. 27J1103, and JSPS fellowship under the No. 15J08654. The authors thank Dr. Bilan Zhu, Dr. Truyen Van Phan, Mr. Cuong Tuan Nguyen, Mr. Hai Dai Nguyen, Mr. Khanh Minh Phan, and other members of Nakagawa Laboratory for availing the symbol recognizer, giving their helpful comments and suggestions. We employ the CROHME 2013, 2014 databases and we are indebted to the people for availing the databases.

Author information

Authors and Affiliations

  1. Department of Computer and Information Sciences, Tokyo University of Agriculture and Technology, 2-24-16, Naka-cho, Koganei-shi, Tokyo, 184-8588, Japan

    Anh Duc Le & Masaki Nakagawa

Authors
  1. Anh Duc Le
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Masaki Nakagawa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Masaki Nakagawa.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Le, A.D., Nakagawa, M. A system for recognizing online handwritten mathematical expressions by using improved structural analysis. IJDAR 19, 305–319 (2016). https://doi.org/10.1007/s10032-016-0272-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10032-016-0272-4

Keywords

Search

Navigation