A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets

  • Scott MacLean
  • George Labahn
Original Paper


We present a new approach for parsing two-dimensional input using relational grammars and fuzzy sets. A fast, incremental parsing algorithm is developed, motivated by the two-dimensional structure of written mathematics. The approach reports all identifiable parses of the input. The parses are represented as a fuzzy set, in which the membership grade of a parse measures the similarity between it and the handwritten input. To identify and report parses efficiently, we adapt and apply existing techniques such as rectangular partitions and shared parse forests, and introduce new ideas such as relational classes and interchangeability. We also present a correction mechanism that allows users to navigate parse results and choose the correct interpretation in case of recognition errors or ambiguity. Such corrections are incorporated into subsequent incremental recognition results. Finally, we include two empirical evaluations of our recognizer. One uses a novel user-oriented correction count metric, while the other replicates the CROHME 2011 math recognition contest. Both evaluations demonstrate the effectiveness of our proposed approach.


Relational Classis Parse Tree Parse Table Membership Grade Terminal Symbol 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Álvaro, F., Sánchez, J.-A., Benedí, J.-M.: Recognition of printed mathematical expressions using two-dimensional stochastic context-free grammars. In: Proceedings of the International Conference on Document Analysis and Recognition, pp. 1225–1229 (2011)Google Scholar
  2. 2.
    Anderson, R.H.: Syntax-directed Recognition of Hand-printed Two-dimensional Mathematics, Ph.D. thesis, Harvard University (1968)Google Scholar
  3. 3.
    Awal, A.-M., Mouchère, H., Viard-Gaudin, C.: The problem of handwritten mathematical expression recognition evaluation. In: Proceedings of the International Conference on Frontiers in Handwriting Recognition, pp. 646–651 (2010)Google Scholar
  4. 4.
    Belaid A., Haton J.-P.: A syntactic approach for handwritten mathematical formula recognition. IEEE Trans. Pattern Anal. Mach. Intell. 6(1), 105–111 (1984)CrossRefGoogle Scholar
  5. 5.
    Blostein D., Grbavec A.: Recognition of mathematical notation. In: Wang, P.S.P., Bunke, H. (eds) Handbook on Optical Character Recognition in Document Analysis, pp. 557–582. World Scientific, Singapore (1996)Google Scholar
  6. 6.
    Blostein D.: Math-literate computers, Calculemus/MKM. In: Carette, J., Dixon, L., Coen, C.S., Watt, S.M. (eds) Lecture Notes in Computer Science, vol. 5625, pp. 2–13. Springer, Berlin (2009)Google Scholar
  7. 7.
    Caraballo S.A., Charniak E.: New figures of merit for best-first probabilistic chart parsing. Comput. Linguist. 24(2), 275–298 (1998)Google Scholar
  8. 8.
    Chan, K.-F., Yeung, D.-Y.: Error detection, error correction and performance evaluation in on-line mathematical expression recognition. In: On-Line Mathematical Expression Recognition, Pattern Recognition (1999)Google Scholar
  9. 9.
    Chan K.-F., Yeung D.-Y.: Mathematical expression recognition: a survey. Int. J. Doc. Anal. Recogn. 3, 3–15 (1999)CrossRefGoogle Scholar
  10. 10.
    Chou, P.A.: Recognition of equations using a two-dimensional stochastic context-free grammar. In: Proceedings of the SPIE, Visual Communication and Image Processing IV, vol. 1199, pp. 852–863 (1989)Google Scholar
  11. 11.
    Costagliola G., De Lucia A., Orefice S., Tortora G.: Positional grammars: a formalism for lr-like parsing of visual languages. In: Marriott, K., Meyer, B. (eds) Visual Language Theory, pp. 171–192. Springer, New York (1998)CrossRefGoogle Scholar
  12. 12.
    Fitzgerald, J.A., Geiselbrechtinger, F., Kechadi, T.: Mathpad: A fuzzy logic-based recognition system for handwritten mathematics, Document Analysis and Recognition. ICDAR 2007. Ninth International Conference on, vol. 2, Sept. 2007, pp. 694–698 (2007)Google Scholar
  13. 13.
    Garain U., Chaudhuri B.B.: Recognition of online handwritten mathematical expressions. Syst. Man. Cybern. Part B: Cybern. IEEE Trans. 34(6), 2366–2376 (2004)Google Scholar
  14. 14.
    Garain U., Chaudhuri B.: A corpus for ocr research on mathematical expressions. Int. J. Doc. Anal. Recognit. 7(4), 241–259 (2005)CrossRefGoogle Scholar
  15. 15.
    Grune D., Jacobs C.J.H.: Parsing Techniques: A Practical Guide, 2 edn. Springer, Berlin (2008)Google Scholar
  16. 16.
    Hull, J.: Recognition of mathematics using a two-dimensional trainable context-free grammar, Master’s thesis, Massachusetts Institute of Technology (1996)Google Scholar
  17. 17.
    Labahn, G., Lank, E., MacLean, S., Marzouk, M., Tausky, D.: Mathbrush: a system for doing math on pen-based devices. In: The Eighth IAPR Workshop on Document Analysis Systems (DAS), pp. 599–606 (2008)Google Scholar
  18. 18.
    Bernard L.: Towards a Uniform Formal Framework for Parsing, Current Issues in Parsing Technology, pp. 153–171. Kluwer, Dordrecht (1991)Google Scholar
  19. 19.
    Laviola, Jr. J.J.: Mathematical Sketching: A New Approach to Creating and Exploring Dynamic Illustrations, Ph.D. thesis, Brown University (2005)Google Scholar
  20. 20.
    Lee E.T., Zadeh L.A.: Note on fuzzy languages. In: Klir, G.J., Yuan, B. (eds) Fuzzy Sets, Fuzzy Logic, and Fuzzy Systems, pp. 69–82. World Scientific, Singapore (1996)Google Scholar
  21. 21.
    Li, C., Miller, T.S., Zeleznik, R.C., LaViola Jr. J.J.: Algosketch: Algorithm sketching and interactive computation. In: Proceedings of Sketch-Based Interfaces and Modeling (2008)Google Scholar
  22. 22.
    Liang, P., Narasimhan, M., Shilman, M., Viola, P.: Efficient geometric algorithms for parsing in two dimensions, ICDAR ’05: Proceedings of the Eighth International Conference on Document Analysis and Recognition (Washington, DC, USA), IEEE Computer Society, pp. 1172–1177 (2005)Google Scholar
  23. 23.
    Liu, W., Zhang, L., Tang, L., Dori, D.: Cost evaluation of interactively correcting recognized engineering drawings, Lecture Notes in Computer Science, no. 1941, pp. 329–334 (2000)Google Scholar
  24. 24.
    MacLean S., Labahn G., Lank E., Marzouk M., Tausky D.: Grammar-based techniques for creating ground-truthed sketch corpora. Int. J. Doc. Anal. Recogn. 14, 65–74 (2011)CrossRefGoogle Scholar
  25. 25.
    MacLean, S.: Tools for the efficient generation of hand-drawn corpora based on context-free grammars. In: Third International Workshop on Pen-Based Mathematics Computing, (2009)
  26. 26.
    MacLean, S., Labahn, G.: Elastic matching in linear time and constant space. In: Proceedings of Ninth IAPR International Workshop on Document Analysis Systems, (Short paper), pp. 551–554 (2010)Google Scholar
  27. 27.
    MacLean, S., Labahn, G.: Recognizing handwritten mathematics via fuzzy parsing, Technical Report CS-2010-13, School of Computer Science, University of Waterloo (2010)Google Scholar
  28. 28.
    Manning C.D., Schuetze H.: Foundations of Statistical Natural Language Processing. The MIT Press, Cambridge (1999)zbMATHGoogle Scholar
  29. 29.
    Marriott K., Meyer B., Wittenburg K.B.: A survey of visual language specification and recognition. In: Marriott, K., Meyer, B. (eds) Visual Language Theory, pp. 5–85. Springer, New York (1998)CrossRefGoogle Scholar
  30. 30.
    Miller, E.G., Viola, P.A.: Ambiguity and constraint in mathematical expression recognition. In: Proceedings of the Fifteenth National/tenth Conference on Artificial Intelligence/Innovative Applications of Artificial Intelligence, pp. 784–791 (1998)Google Scholar
  31. 31.
    Mouchère, H., Viard-Gaudin, C., Garain, U., Kim, D.H., Kim, J.H.: Crohme2011: Competition on recognition of online handwritten mathematical expressions. In: Proceedings of the 11th International Conference on Document Analysis and Recognition (2011)Google Scholar
  32. 32.
    Rhee T.H., Kim J.H.: Efficient search strategy in structural analysis for handwritten mathematical expression recognition. Pattern Recogn. 42(12), 3192–3201 (2009)zbMATHCrossRefGoogle Scholar
  33. 33.
    Rutherford, I.: Structural analysis for pen-based math input systems, Master’s thesis, David R. Cheriton School of Computer Science, University of Waterloo (2005)Google Scholar
  34. 34.
    Sain K., Dasgupta A., Garain U.: Emers: a tree matching-based performance evaluation of mathematical expression recognition systems. IJDAR 14(1), 75–85 (2011)CrossRefGoogle Scholar
  35. 35.
    Shi, Y., Li, H., Soong, F.K.: A unified framework for symbol segmentation and recognition of handwritten mathematical expressions. In: Ninth International Conference on Document Analysis and Recognition, pp. 854–858 (2007)Google Scholar
  36. 36.
    Tomita M.: Parsing 2-dimensional languages. In: Tomita, M. (eds) Current Issues in Parsing Technology, pp. 277–289. Kluwer, Dordrecht (1991)CrossRefGoogle Scholar
  37. 37.
    Toyota, S., Uchida, S., Suzuki, M.: Structural analysis of mathematical formulae with verification based on formula description grammar. Doc. Anal. Syst. VII:153–163 (2006)Google Scholar
  38. 38.
    Unger S.H.: A global parser for context-free phrase structure grammars. Commun. ACM 11, 240–247 (1968)MathSciNetCrossRefGoogle Scholar
  39. 39.
    Winkler, H.-J., Fahrner, H., Lang, M.: A soft-decision approach for structural analysis of handwritten mathematical expressions. In: Proceedings of International Conference on Acoustics, Speech and Signal Processing, pp. 2459–2462 (1995)Google Scholar
  40. 40.
    Zadeh L.A.: Fuzzy sets. Inf. Control 8, 338–353 (1965)MathSciNetzbMATHCrossRefGoogle Scholar
  41. 41.
    Zanibbi R., Blostein D., Cordy J.R.: Recognizing mathematical expressions using tree transformation. IEEE Trans. Pattern Anal. Mach. Intell. 24(11), 1455–1467 (2002)CrossRefGoogle Scholar
  42. 42.
    Zhang, L., Blostein, D., Zanibbi, R.: Using fuzzy logic to analyze superscript and subscript relations in handwritten mathematical expressions. In: Proceedings of Eighth International Conference on Document Analysis and Recognition, vol. 2, pp. 972–976 (2005)Google Scholar

Copyright information

© Springer-Verlag 2012

Authors and Affiliations

  1. 1.Cheriton School of Computer ScienceUniversity of WaterlooWaterlooCanada

Personalised recommendations