Accommodating Phonetic Word Variations Through Generated Confusion Pairs for Hinglish Handwritten Text Recognition

  • Soumyajit Mitra
  • Vikrant Singh
  • Pragya Paramita Sahu
  • Viswanath Veera
  • Shankar M. Venkatesan
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10859)


On-line handwriting recognition has seen major strides in the past years, especially with the advent of deep learning techniques. Recent work has seen the usage of deep networks for sequential classification of unconstrained handwriting recognition task. However, the recognition of “Hinglish” language faces various unseen problems. Hinglish is a portmanteau of Hindi and English, involving frequent code-switching between the two languages. Millions of Indians use Hinglish as a primary mode of communication, especially across social media. However, being a colloquial language, Hinglish does not have a fixed rule set for spelling and grammar. Auto-correction is an unsuitable solution as there is no correct form of the word, and all the multiple phonetic variations are valid. Unlike the advantage that keyboards provide, recognizing handwritten text also has to overcome the issue of mis-recognizing similar looking alphabets. We propose a comprehensive solution to overcome this problem of recognizing words with phonetic spelling variations. To our knowledge, no work has been done till date to recognize Hinglish handwritten text. Our proposed solution shows a character recognition accuracy of 94% and word recognition accuracy of 72%, thus correctly recognizing the multiple phonetic variations of any given word.


Handwriting recognition Colloquial language recognition Phonetic spelling variations Transliteration 


  1. 1.
    Bahlmann, C., Burkhardt, H.: The writer independent online handwriting recognition system frog on hand and cluster generative statistical dynamic time warping. IEEE Trans. Pattern Anal. Mach. Intell. 26(3), 299–310 (2004)CrossRefGoogle Scholar
  2. 2.
    Bahlmann, C., Haasdonk, B., Burkhardt, H.: Online handwriting recognition with support vector machines-a kernel approach. In: Proceedings of the Eighth International Workshop on Frontiers in Handwriting Recognition, 2002, pp. 49–54. IEEE (2002)Google Scholar
  3. 3.
    Bharath, A., Madhvanath, S.: Hidden Markov Models for online handwritten Tamil word recognition. In: ICDAR, pp. 506–510. IEEE (2007)Google Scholar
  4. 4.
    Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 369–376. ACM (2006)Google Scholar
  5. 5.
    Graves, A., Liwicki, M., Bunke, H., Schmidhuber, J., Fernández, S.: Unconstrained on-line handwriting recognition with recurrent neural networks. In: Advances in Neural Information Processing Systems, pp. 577–584 (2008)Google Scholar
  6. 6.
    Graves, A., Liwicki, M., Fernández, S., Bertolami, R., Bunke, H., Schmidhuber, J.: A novel connectionist system for unconstrained handwriting recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31(5), 855–868 (2009)CrossRefGoogle Scholar
  7. 7.
    Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 18(5), 602–610 (2005)CrossRefGoogle Scholar
  8. 8.
    Hu, J., Lim, S.G., Brown, M.K.: Writer independent on-line handwriting recognition using an HMM approach. Pattern Recogn. 33(1), 133–147 (2000)CrossRefGoogle Scholar
  9. 9.
    Jhamtani, H., Bhogi, S.K., Raychoudhury, V.: Word-level language identification in bi-lingual code-switched texts. In: PACLIC, pp. 348–357 (2014)Google Scholar
  10. 10.
    Plamondon, R., Srihari, S.N.: Online and off-line handwriting recognition: a comprehensive survey. IEEE Trans. Pattern Anal. Mach. Intell. 22(1), 63–84 (2000)CrossRefGoogle Scholar
  11. 11.
    Baldauf, S.: A Hindi-English jumble, spoken by 350 million (2004).
  12. 12.
    Tappert, C.C., Suen, C.Y., Wakahara, T.: The state of the art in online handwriting recognition. IEEE Trans. Pattern Anal. Mach. Intell. 12(8), 787–808 (1990)CrossRefGoogle Scholar
  13. 13.
    Zhou, X.D., Yu, J.L., Liu, C.L., Nagasaki, T., Marukawa, K.: Online handwritten Japanese character string recognition incorporating geometric context. In: Ninth International Conference on Document Analysis and Recognition, ICDAR 2007, vol. 1, pp. 48–52. IEEE (2007)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Soumyajit Mitra
    • 1
  • Vikrant Singh
    • 1
  • Pragya Paramita Sahu
    • 1
  • Viswanath Veera
    • 1
  • Shankar M. Venkatesan
    • 1
  1. 1.Samsung R&D Institute India - BangaloreBangaloreIndia

Personalised recommendations