Skip to main content
SpringerLink
Log in

MDA-Network: Mask and Dual Attention Network for Handwritten Mathematical Expression Recognition

  • Conference paper
  • First Online:
Computer Supported Cooperative Work and Social Computing (ChineseCSCW 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1330))

Abstract

Building a system for automatic handwritten mathematical expressions recognition (HMER) has received considerable attention for its extensive applications. However, HMER remains challenging due to its own many characteristics such as the ambiguity of handwritten symbol, the two-dimensional characteristics of expression structure, and a large amount of context information. Inspired by research on machine translation and image caption, we proposed an Encoder-Decoder structure to recognize the handwritten mathematical expression. Encoder based dual attention is used to extract the features of the expression image and attention-decoder achieves symbol recognition and structural analysis. The mask information is added to the input data allows the model to better focus on the region of interest. In order to verify the effectiveness of our method, we train the model on the CROHME-2016 train set and use the CROHME-2014 test set as the validation set, the CROHME-2016 test set as the test set. The experimental results show that our method is greatly improved compared with other recognition methods, achieved respectively 47.49% and 45.10% ExpRate in the two test sets.

Supported by Transformation and industrialization demonstration of marine scientific and technological achievements in Xiamen Marine and Fisheries Bureau (No. 18CZB033HJ11).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 159.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Zhang, J., Du, J., Dai, L.: Multi-scale attention with dense encoder for handwritten mathematical. In: Conference on Pattern Recognition (ICPR), pp. 2245–2250. IEEE, Beijing (2018)

    Google Scholar 

  2. Mouchere, H., Viard-Gaudin, C., Zanibbi, R.: ICFHR 2014 competition on recognition of on-line handwritten mathematical expressions. In: 2014 14th International Conference on Frontiers in Handwriting Recognition, pp. 791–796. IEEE, Hersonissos, Greece (2014)

    Google Scholar 

  3. Long, C., Zhang, H., Xiao, J.: SCA-CNN: spatial and channel-wise attention in convolutional networks for image captioning. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6298–6306. IEEE, Honolulu (2017)

    Google Scholar 

  4. Le, A.D., Nakagawa, M.: Training an end-to-end system for handwritten mathematical expression recognition by generated patterns. In: 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), pp. 1056–1061. IEEE, Kyoto (2017)

    Google Scholar 

  5. Vinyals, O., Toshev, A., Bengio, S.: Show and tell: a neural image caption generator. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3156–3164. IEEE, Boston (2015)

    Google Scholar 

  6. Xu, K., Ba, J., Kiros, R.: Show, attend and tell: neural image caption generation with visual attention. In: Proceedings of the 32nd International Conference on Machine Learning, pp. 2048–2057(2015)

    Google Scholar 

  7. Zhang, J., Du, J., Zhang, S.: Watch, attend and parse: an end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recognit. 71, 196–206 (2017)

    Google Scholar 

  8. Jie, H., Li, S., Gang, S.: Squeeze-and-excitation networks. In: 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7132–7141. IEEE, Salt Lake City (2018)

    Google Scholar 

  9. Tu, Z., Lu, Z., Liu, Y.: Modeling coverage for neural machine translation. In: 54th Annual Meeting of the Association-for-Computational-Linguistics (ACL), pp. 76–85. Berlin (2016)

    Google Scholar 

  10. Mouchère, H., Viard-Gaudin, C., Zanibbi, R.: ICFHR2016 CROHME: competition on recognition of online handwritten mathematical expressions. In: 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), Shenzhen, pp. 607–612 (2016)

    Google Scholar 

  11. He, K., Zhang, X., Ren, S.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. IEEE, Seattle (2016)

    Google Scholar 

  12. Huang, G., Liu, Z., Laurens, V.D.M.: Densely connected convolutional networks. In: 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4700–4708. IEEE, Honolulu (2017)

    Google Scholar 

  13. Bhunia, A.K., Bhowmick, A., Bhunia, A.K.: Handwriting trajectory recovery using end-to-end deep encoder-decoder network. In: 24th International Conference on Pattern Recognition (ICPR), pp. 3639–3644. IEEE, Beijing (2018)

    Google Scholar 

  14. Chan, K.F., Yeung, D.Y.: Mathematical expression recognition: a survey. Int. J. Document Anal. Recognit. (ICDAR), 3–15 (2000). https://doi.org/10.1007/PL00013549

  15. Chen, X., Ma, L., Jiang, W.: Regularizing RNNs for caption generation by reconstructing the past with the present. In: 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7995–8003. IEEE, Salt Lake City (2018)

    Google Scholar 

  16. Le, A.D., Indurkhya, B., Nakagawa, M.: Pattern generation strategies for improving recognition of handwritten mathematical expressions. Pattern Recognit. Lett.128, 255–262 (2019)

    Google Scholar 

  17. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks, pp. 249–256 (2010)

    Google Scholar 

  18. Woo, S., Park, J., Lee, J.Y.: CBAM: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19(2018)

    Google Scholar 

  19. Alvaro, F., Sánchez, J., Benedí, J.: Recognition of on-line handwritten mathematical expressions using 2D stochastic context free grammars and Hidden Markov models. Pattern Recognit. Lett. 35, 58–67 (2014)

    Google Scholar 

  20. Alvaro, F., Sánchez, J., Benedí, J.: An integrated grammar based approach for mathematical expression recognition. Pattern Recognit. 51, 135–147 (2016)

    Google Scholar 

  21. Zanibbi, R., Blostein, D., Cordy, J.R.: Recognizing mathematical expressions using tree transformation. IEEE Trans. Pattern Anal. Mach. Intell. 24(11), 1455–1467 (2002)

    Google Scholar 

  22. MacLean, S., Labahn, G.: A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets. Int. J. Document Anal. Recognit. 139–163 (2013). https://doi.org/10.1007/s10032-012-0184-x

  23. Mouchere, H., Zanibbi, R., Garain, U.: Advancing the state-of-the-art for handwritten math recognition. Int. J. Document Anal. Recognit. 173–189 (2016)

    Google Scholar 

  24. Zanibbi, R., Blostein, D.: Recognition and retrieval of mathematical expressions. Int. J. Document Anal. Recognit. 15(4), 331–357 (2012). https://doi.org/10.1007/s10032-011-0174-4

  25. Chan, K.F., Yeung, D.Y.: Error detection, error correction and performance evaluation in on-line mathematical expression recognition. Pattern Recognit. 34(8), 1671–1684 (2001)

    Article  Google Scholar 

  26. Dauphin, Y.N., Fan, A., Auli, M.: Language modeling with gated convolutional networks. In: Proceedings of the 34th International Conference on Machine Learning, pp. 70:933–941(2017)

    Google Scholar 

  27. Mouchere, H., Zanibbi, R., Garain, U.: Advancing the state-of-the-art for handwritten math recognition. Inte. J. Document Anal. Recognit. 19(2), 173–189 (2016)

    Article  Google Scholar 

  28. Zhang, J., Du, J., Dai, L.: A GRU-based encoder-decoder approach with attention for online handwritten mathematical expression recognition. In: 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), pp. 902–907. IEEE, Kyoto (2017)

    Google Scholar 

  29. Anderson, R.H.: Syntax-directed recognition of hand-printed two-dimensional mathematics. In: Symposium on Interactive Systems for Experimental Applied Mathematics: Proceedings of the Association for Computing Machinery Inc., Symposium, pp. 436–459 (1967)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Xiamen University, Xiamen, 361005, China

    Jian Hu, Pan Zhou, Shirong Liao, Shaozi Li & Songzhi Su

  2. Ropeok Technology Group Co., Ltd., Xiamen, China

    Songjian Su

Authors
  1. Jian Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pan Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shirong Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shaozi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Songjian Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Songzhi Su
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Songzhi Su .

Editor information

Editors and Affiliations

  1. Shandong University, Jinan, China

    Yuqing Sun

  2. Guangdong University of Technology, Guangzhou, China

    Dongning Liu

  3. Shenzhen University, Shenzhen, China

    Hao Liao

  4. Tongji University, Shanghai, China

    Hongfei Fan

  5. University of Shanghai for Science and Technology, Shanghai, China

    Liping Gao

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hu, J., Zhou, P., Liao, S., Li, S., Su, S., Su, S. (2021). MDA-Network: Mask and Dual Attention Network for Handwritten Mathematical Expression Recognition. In: Sun, Y., Liu, D., Liao, H., Fan, H., Gao, L. (eds) Computer Supported Cooperative Work and Social Computing. ChineseCSCW 2020. Communications in Computer and Information Science, vol 1330. Springer, Singapore. https://doi.org/10.1007/978-981-16-2540-4_15

Download citation

  • DOI: https://doi.org/10.1007/978-981-16-2540-4_15

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-16-2539-8

  • Online ISBN: 978-981-16-2540-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation