Advertisement

Stroke order normalization for improving recognition of online handwritten mathematical expressions

  • Anh Duc LeEmail author
  • Hai Dai Nguyen
  • Bipin Indurkhya
  • Masaki Nakagawa
OriginalPaper

Abstract

We present a technique based on stroke order normalization for improving recognition of online handwritten mathematical expressions (ME). The stroke order dependent system has less time complexity than the stroke order free system, but it must incorporate special grammar rules to cope with stroke order variations. The stroke order normalization technique solves this problem and also the problem of unexpected stroke order variations without increasing the time complexity of ME recognition. In order to normalize stroke order, the XY cut method is modified since its original form causes problems when structural components in ME overlap. First, vertically ordered strokes are located by detecting vertical symbols and their upper/lower components, which are treated as MEs and reordered recursively. Second, unordered strokes on the left side of the vertical symbols are reordered as horizontally ordered strokes. Third, the remaining strokes are reordered recursively. The horizontally ordered strokes are reordered from left to right, and the vertically ordered strokes are reordered from top to bottom. Finally, the proposed stroke order normalization is combined with the stroke order dependent ME recognition system. The evaluations on the CROHME 2014 database show that the ME recognition system incorporating the stroke order normalization outperforms all other systems that use only CROHME 2014 for training while the processing time is kept low.

Keywords

Recognition of online handwritten mathematical expressions XY cut Stroke order normalization 

Notes

References

  1. 1.
    Chan, K., Yeung, D.: Mathematical expression recognition: a survey. Int. J. Doc. Anal. Recognit. 3, 3–15 (2000)CrossRefGoogle Scholar
  2. 2.
    Zanibbi, R., Blostein, D.: Recognition and retrieval of mathematical expressions. Int. J. Doc. Anal. Recognit. 15, 331–357 (2012)CrossRefGoogle Scholar
  3. 3.
    Mouchere, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: ICFHR 2014 competition on recognition of on-line handwritten mathematical expressions (CROHME 2014). In: International Conference Frontiers in Handwriting Recognition, pp. 791–796 (2014)Google Scholar
  4. 4.
    Lehmberg, S., Winkler, H.J., Lang, M.: A soft-decision approach for symbol segmentation within handwritten mathematical expressions, International conference on acoustics, speech, and signal processing, vol. 6, pp. 3434–3437, Atlanta (1996)Google Scholar
  5. 5.
    Toyozumi, K., et al.: A study of symbol segmentation method for handwritten mathematical formula recognition using mathematical structure information. In: International Conference on Pattern Recognition, vol. 2, pp. 630–633, Cambridge (2004)Google Scholar
  6. 6.
    Hu, L., Zanibbi, R.: Segmenting handwritten math symbols using adaboost and multi-scale shape context features. In: International Conference on Document Analysis and Recognition, pp. 1180–1184, Washington (2013)Google Scholar
  7. 7.
    MacLean, S., Labahn, G.: Elastic matching in linear time and constant space. In: IAPR Workshop on Document Analysis Systems (2010)Google Scholar
  8. 8.
    Hu, L., Zanibbi, R.: HMM-based recognition of online hand-written mathematical symbols using segmental K-means initialization and a modified pen-up/down feature. In: International Conference on Document Analysis and Recognition, pp. 457–462, Beijing (2011)Google Scholar
  9. 9.
    Alvaro, F., Sanchez, J.A., Benedi, J.M.: Classification of on-line mathematical symbols with hybrid features and recurrent neural networks, International Conference on Document Analysis and Recognition, pp. 1012–1016, Washington (2013)Google Scholar
  10. 10.
    Davila, K.M., Ludi, S., Zanibbi R.: Using off-line features and synthetic data for on-line handwritten math symbol recognition, International Conference on Frontiers in Handwriting Recognition, pp. 323–328, Crete (2014)Google Scholar
  11. 11.
    Garain, U., Chaudhuri, B.B.: Recognition of online handwritten mathematical expressions. IEEE Trans. Syst. Man Cybern. Part B Cybern. 34, 2366–2376 (2004)CrossRefGoogle Scholar
  12. 12.
    Alvaro, F., Sanchez, J.A., Benedi, J.M.: Offline features for classifying handwritten math symbols with recurrent neural networks. In: International Conference on Pattern Recognition, pp. 2944–2949, Stockholm (2014)Google Scholar
  13. 13.
    Nguyen, H.D., Le, A.D., Nakagawa, M.: Deep neural network for recognizing online handwritten mathematical symbols. In: IAPR Asian Conference on Pattern Recognition, pp. 121–125 (2015)Google Scholar
  14. 14.
    Nguyen, H.D., Le, A.D., Nakagawa, M.: Recognition of online handwritten math symbols using deep neural networks, IEICE Transactions on Information and Systems, vol. E99.D, pp. 3110–3118 (2016)Google Scholar
  15. 15.
    Yamamoto, R., Sako, S., Nishimoto, T., Sagayama, S.: Online recognition of handwritten mathematical expressions based on stroke-based stochastic context-free grammar. In: International Workshop on Frontiers in Handwriting Recognition, pp. 249–254, La Baule, France (2006)Google Scholar
  16. 16.
    Simistira, F., Katsouros, V., Carayannis, G.: Recognition of online handwritten mathematical formulas using probabilistic SVMs and stochastic context free grammars. Pattern Recognit. Lett. 53, 85–92 (2015)CrossRefGoogle Scholar
  17. 17.
    Le, A.D., Nakagawa, M.: A system for recognizing online handwritten mathematical expressions by using improved structural analysis. Int. J. Doc. Anal. Recognit. 19, 305–319 (2016)CrossRefGoogle Scholar
  18. 18.
    Le, A.D., Phan, T.V., Nakagawa, M.: A system for recognizing online handwritten mathematical expressions and improvement of structure analysis. In: IAPR Workshop on Document Analysis Systems, pp. 51–55 (2014)Google Scholar
  19. 19.
    MacLean, S., Labahn, G.: A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets. Int. J. Doc. Anal. Recognit. 16, 139–163 (2013)CrossRefGoogle Scholar
  20. 20.
    Alvaro, F., Sanchez, J., Benedi, J.: Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden markov models. Pattern Recognit. Lett. 35, 58–67 (2014)CrossRefGoogle Scholar
  21. 21.
    Lee, H.-J., Wang Lee, J.-S.: Design of a mathematical expression understanding system. Pattern Recognit. Lett. 18, 289–298 (1997)CrossRefGoogle Scholar
  22. 22.
    Zanibbi, R., Blostein, D., Cordy, J.R.: Recognizing mathematical expressions using tree transformation. IEEE Trans. Pattern Anal. Mach. Intell. 24(11), 1455–1467 (2002)CrossRefGoogle Scholar
  23. 23.
    Nagy, G., Seth, S.: Hierarchical representation of optically scanned documents. In: International Conference on Pattern Recognition, pp. 347–349, Montreal, Canada (1984)Google Scholar
  24. 24.
    Meunier, J.: Optimized XY-cut for determining a page reading order. In: International Conference on Document Analysis and Recognition, pp. 347–351, Seoul, Korea (2005)Google Scholar
  25. 25.
    Le, A.D., Nguyen, H.D., Nakagawa, M.: Modified XY cut for re-ordering strokes of online handwritten mathematical expressions. In: IAPR Workshop on Document Analysis Systems, pp. 233–238, Greece (2016)Google Scholar
  26. 26.
    Eto, Y., Suzuki, M.: Mathematical formula recognition using virtual link network. In: International Conference on Document Analysis and Recognition, pp. 430–437, USA (2001)Google Scholar
  27. 27.
    Aly, W., Uchida, S., Suzuki, M.: Identifying subscripts and superscripts in mathematical documents. Math. Comput. Sci. 2, 195–209 (2008)CrossRefzbMATHGoogle Scholar
  28. 28.
    Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., Bengio, Y.: Show, attend and tell: neural image caption generation with visual attention. In: International Conference on Machine Learning, vol. 37, pp 2048–2057 (2015)Google Scholar
  29. 29.
    Luong, T., Pham, H., Manning, C.: Effective approaches to attention-based neural machine translation. In: The 2015 Conference on Empirical Methods in Natural Language Processing (2015)Google Scholar
  30. 30.
    Deng, Y., Kanervisto, A., Ling, J., Rush, A.: Image-to-Markup Generation with Coarse-to-Fine Attention, pp. 980–989. ICML, Pittsburgh (2017)Google Scholar
  31. 31.
    Zhang, J., Du, J., Zhang, S., Liu, D., Hu, Y., Hu, J., Wei, S., Dai, L.: Watch, attend and parse: an end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recognit. 71, 196–206 (2017)CrossRefGoogle Scholar
  32. 32.
    Le, A.D., Nakagawa, M.: Training an end-to-end system for handwritten mathematical expression recognition by generated patterns. In: International Conference on Document Analysis and Recognition, pp. 1056–1061 (2017)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  • Anh Duc Le
    • 1
    • 2
    Email author
  • Hai Dai Nguyen
    • 3
  • Bipin Indurkhya
    • 4
  • Masaki Nakagawa
    • 5
  1. 1.Division of Algorithms and Technologies for Networks AnalysisTon Duc Thang UniversityHo Chi Minh CityVietnam
  2. 2.Faculty of Information TechnologyTon Duc Thang UniversityHo Chi Minh CityVietnam
  3. 3.Bioinformatics Center, Institute for Chemical ResearchKyoto UniversityKyotoJapan
  4. 4.Departments of Computer Science and Cognitive ScienceJagiellonian University CracowKrakówPoland
  5. 5.Department of Computer and Information SciencesTokyo University of Agriculture and TechnologyTokyoJapan

Personalised recommendations