Abstract
This paper presents an augmented incremental recognition method for online handwritten mathematical expressions (MEs). If an ME is recognized after all strokes are written (batch recognition), the waiting time increases significantly when the ME becomes longer. On the other hand, the pure incremental recognition method recognizes an ME whenever a new single stroke is input. It shortens the waiting time but degrades the recognition rate due to the limited context. Thus, we propose an augmented incremental recognition method that not only maintains the advantage of the two methods but also reduces their weaknesses. The proposed method has two main features: one is to process the latest stroke, and the other is to find the erroneous segmentations and recognitions in the recent strokes and correct them. In the first process, the segmentation and the recognition by Cocke–Younger–Kasami (CYK) algorithm are only executed for the latest stroke. In the second process, all the previous segmentations are updated if they are significantly changed after the latest stroke is input, and then, all the symbols related to the updated segmentations are updated with their recognition scores. These changes are reflected in the CYK table. In addition, the waiting time is further reduced by employing multi-thread processes. Experiments on our dataset and the CROHME datasets show the effectiveness of this augmented incremental recognition method, which not only maintains recognition rate even compared with the batch recognition method but also reduces the waiting time to a very small level.
Similar content being viewed by others
References
Anderson, R.H.: Syntax-directed recognition of hand-printed two-dimensional mathematics. In: Symposium on Interactive Systems for Experimental Applied Mathematics: Proceedings of the Association for Computing Machinery Inc. Symposium, pp. 436-459, Washington, USA (1967)
Chang, S.K.: A method for the structural analysis of two-dimensional mathematical expressions. Int. J. Inf. Sci. 2(3), 253–272 (1970)
lvaro, F., Snchez, Bened, J.: Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden Markov models. Pattern Recognit. Lett. 31, 58–67 (2014)
Awal, A.M., Mouchre, H., Viard-Gaudin, C.: A global learning approach for an online handwritten mathematical expression recognition system. Pattern Recognit. Lett. 35, 68–77 (2012)
Yamamoto, R., Sako, S., Nishimoto, T., Sagayama, S.: Online recognition of handwritten mathematical expressions based on stroke-based stochastic context-free grammar. In: Proceedings of the 7th International Conference on Document Analysis and Recognition, vol. 1, pp. 249-254, Edinburgh, UK (2006)
Simistira, F., Katsouros, V., Carayannis, G.: Recognition of online handwritten mathematical formulas using probabilistic SVMs and stochastic context free grammars. Pattern Recognit. Lett. 53, 85–92 (2015)
Le, A.D., Nakagawa, M.: A system for recognizing online handwritten mathematical expressions by using improved structural analysis. Int. J. Doc. Anal. Recognit. 19(4), 305–319 (2016)
Okamoto, M., Miao, B.: Recognition of mathematical expressions by using the layout structure of symbols. In: Proceeding of the 1st International Conference on Document Analysis and Recognition, pp. 242–250, Saint-Malo, France (1991)
Zanibbi, R., Blostein, D., Cordy, J.R.: Recognizing mathematical expressions using tree transformation. IEEE Trans. Pattern Anal. Mach. Intell. 24(11), 1455–1467 (2002)
Hu, L., Zanibbi, R.: MST-based Visual Parsing of Online Handwritten Mathematical Expressions. In: Proceedings of the 15th International Conference on Frontiers in Handwriting Recognition, pp. 337–342, Shenzhen, China (2016)
Julca-Aguilar, F., Mouchre, H., Viard-Gaudin, C., Hirata, N.S.T.: Top-Down Online Handwritten Mathematical Expression Parsing with Graph Grammar. In: Proceedings of the 20th Iberoamerican Congress on Pattern Recognition, pp. 444–451, Montevideo, Uruguay (2015)
Garain, U., Chaudhuri, B.B.: Recognition of online handwritten mathematical expressions. IEEE Trans. Syst. Man Cybern. B Cybern. 34(6), 2366–2376 (2004)
Le, A.D., Nakagawa, M.: A system for recognizing online handwritten mathematical expressions by using improved structural analysis. Int. J. Doc. Anal. Recognit. 19(4), 305–319 (2016)
Zhang, J., Du, J., Zhang, S., Liu, D., Hu, Y., Hu, J., Wei, S., Dai, L.: Watch, attend and parse: an end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recognit. Lett. 71, 196–206 (2017)
MacLean, S., Labahn, G.: A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets. Int. J. Doc. Anal. Recognit. 16(2), 139–163 (2013)
Unger, S.H.: A global parser for context-free phrase structure grammars. Commun. ACM 11(4), 240–247 (1968)
Predovic, G., Abdulkader, A., Dresevic, B., Viola, P. A, Vukosavljevic, M.: Recognition of mathematical expressions. U.S. Patent US8009915 B2 (2011)
Vuong, B.Q., Hui, S.C., He, Y.: Progressive structural analysis for dynamic recognition of on-line handwritten mathematical expressions. Pattern Recognit. Lett. 29(5), 647–655 (2008)
Phan, K.M., Nguyen, C.T., Le, A.D., Nakagawa, M.: An incremental recognition method for online handwritten mathematical expressions. In: Proceedings of the 3rd IAPR Asian Conference on Pattern Recognition, pp. 171–175, Kuala Lumpur, Malaysia (2015)
Phan, K.M., Le, A.D., Nakagawa, M.: Semi-incremental recognition of online handwritten mathematical expressions. In: Proceedings of the 15th International Conference on Frontiers in Handwriting Recognition, pp. 258–264, Shenzhen, China (2016)
Nakagawa, M., Machii, K., Kato, N., Souya, T.: Lazy recognition as a principle of pen interfaces. In: Proceedings of the ACM INTERCHI, pp. 89–90, Amsterdam, Netherlands (1993)
Le, A.D., Nakagawa, M.: Comparison of parsing algorithms for recognizing online handwritten mathematical expressions. In: Proceedings of the 15th International Conference on Frontiers in Handwriting Recognition, pp. 390–394, Shenzhen, China (2016)
Cocke, J., Schwartz, J.T.: Programming Languages and Their Compilers: Preliminary Notes, 2nd edn. Courant Institute of Mathematical Sciences, New York (1970)
Younger, D.H.: Recognition and parsing of context-free languages in time \(n^3\). Inf. Comput. 10(2), 189–208 (1967)
Kasami, T.: An Efficient Recognition and Syntax-Analysis Algorithm for Context-Free Languages. University of Illinois Coordinated Science Laboratory, Amsterdam (1966)
Zhu, B., Gao, J., Nakagawa, M.: Objective function design for MCE-based combination of on-line and off-line character recognizers for on-line handwritten Japanese text recognition. In: Proceedings of the 11th International Conference on Document Analysis and Recognition, pp. 594–599, Beijing, China (2011)
Mouchre, H., Viard-Gaudin, C., Zanibbi, R., Garain, U., Kim, D.H., Kim, J.H.: ICDAR 2013 CROHME: third international competition on recognition of online handwritten mathematical expressions. In: Proceedings of the 12th International Conference on Document Analysis and Recognition, pp. 1428–1432, Washington, USA (2013)
Mouchre, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: ICFHR 2014 competition on recognition of on-line handwritten mathematical expressions (CROHME 2014). In: Proceedings of the 14th International Conference on Frontiers in Handwriting Recognition, pp. 791–796, Heraklion, Greece (2014)
Mouchre, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: ICFHR2016 CROHME: competition on recognition of online handwritten mathematical expressions. In: Proceedings of the 15th International Conference on Frontiers in Handwriting Recognition, pp. 607–612, Shenzhen, China (2016)
Nielsen, J.: Usability Engineering. Academic Press Inc, Boston (1993)
Shneiderman, B.: Response time and display rate in human performance with computer. ACM Comput. Surv. 16(3), 265–285 (1984)
Miller, R.B.: Response time in man-computer conversational transactions. In: Proceeding of the AFIPS Fall Joint Computer Conference, vol. 33, pp. 267–277, San Francisco, USA (1968)
Liu, C.L., Jaeger, S., Nakagawa, M.: Online recognition of Chinese characters: the state-of-the-art. IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 198–213 (2004)
Plamondon, R., Srihari, S.N.: On-line and off-line handwriting recognition: a comprehensive survey. IEEE Trans. Pattern Anal. Mach. Intell. 22(1), 63–84 (2000)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Phan, K.M., Le, A.D., Indurkhya, B. et al. Augmented incremental recognition of online handwritten mathematical expressions. IJDAR 21, 253–268 (2018). https://doi.org/10.1007/s10032-018-0306-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10032-018-0306-1