Skip to main content
SpringerLink
Log in

Two streams deep neural network for handwriting word recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Handwritten word recognition is one of the hot topics in automatic handwritten text recognition that received a lot of attention in recent years. Unlike character recognition, word recognition deals with considerable variations in word shape and written style. This paper proposes a novel deep model for language-independent handwritten word recognition. The proposed deep structure has two parallel stages for jointly learning character and word-level information. In the character-level stage, a weakly character segmentation method is performed and then applies a series of Long short-term memory (LSTM) layers for character-level representation. The word-level stage employs a series of convolutional layers for the shape and structure representation of the word. These representations are then concatenated and followed by a series of fully connected layers for jointly learning the words and the character-level information. Since the character segmentation is language independent and error-prone, the proposed deep structure only applies weakly separation scheme and does not rely on any character segmentation algorithm. Thus, it effectively utilizes character level representation without bounding on any language model. In the proposed methodology, we use two new data augmentation strategies based on a psychological assumption to increase the model generalization performance. Experimental results on five public datasets including Arabic, English and German languages demonstrate that the proposed deep model has a superior performance to the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Ajmire P (2012) Structural features for character recognition systema review

  2. AlKhateeb JH, Ren J, Jiang J, Ipson SS, Abed HE (2008) Word-based handwritten arabic scripts recognition using dct features and neural network classifier. In: 2008 5th international multi-conference on systems, signals and devices, pp 1–5

  3. AlKhateeb JHY, Ren J, Jiang J, Al-Muhtaseb H (2011) Offline handwritten arabic cursive text recognition using hidden markov models and re-ranking. Pattern Recogn Lett 32:1081–1088

    Article  Google Scholar 

  4. Almazán J, Gordo A, Fornés A, Valveny E (2014) Word spotting and recognition with embedded attributes. IEEE Transactions on Pattern Analysis and Machine Intelligence 36:2552–2566

    Article  Google Scholar 

  5. Amrouch M, Rabi M, Es-Saady Y (2018) Convolutional feature learning and cnn based hmm for arabic handwriting recognition. In: International conference on image and signal processing. Springer, pp 265–274

  6. Azad R, Davami F, Shayegh HR (2014) Recognition of handwritten persian/arabic numerals based on robust feature set and k-nn classifier. arXiv:1407.6492

  7. Babu N, Soumya A (2019) Character recognition in historical handwritten documents – a survey. In: 2019 international conference on communication and signal processing (ICCSP), pp 0299– 0304

  8. Bernardo LS, Quezada A, Munoz R, Maia FM, Pereira CR, Wu W, de Albuquerque VHC (2019) Handwritten pattern recognition for early parkinson’s disease diagnosis. http://www.sciencedirect.com/science/article/pii/S016786551930114X, vol 125, pp 78–84

  9. Bhunia AK, Konwer A, Bhowmick A, Bhunia AK, Roy PP, Pal U (2019) Script identification in natural scene image and video frames using an attention based convolutional-lstm network. Pattern Recogn 85:172–184

    Article  Google Scholar 

  10. Bluche T, Messina RO (2017). In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR), vol 01, pp 646–651

  11. Bluche T, Ney H, Kermorvant C (2013) Tandem hmm with convolutional neural network for handwritten word recognition. In: 2013 IEEE international conference on acoustics, speech and signal processing, pp 2390–2394

  12. Bluche T, Ney H, Kermorvant C (2014) A comparison of sequence-trained deep neural networks and recurrent neural networks optical modeling for handwriting recognition. In: SLSP

  13. Boquera SE, Bleda MJC, Gorbe-Moya J, Zamora-Martínez F (2011) Improving offline handwritten text recognition with hybrid hmm/ann models. IEEE Transactions on Pattern Analysis and Machine Intelligence 33:767–779

    Article  Google Scholar 

  14. Castro D, Bezerra BLD, Valenca M (2018) Boosting the deep multidimensional long-short-term memory network for handwritten recognition systems. In: 2018 16th international conference on frontiers in handwriting recognition (ICFHR), pp 127–132

  15. Chen Z, Wu Y, Yin F, Liu C-L (2017) Simultaneous script identification and handwriting recognition via multi-task learning of recurrent neural networks. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR), vol 01, pp 525–530

  16. Chowdhury A, Vig L (2018) An efficient end-to-end neural model for handwritten text recognition. In: BMVC

  17. Dutta K, Krishnan P, Mathew M, Jawahar CV (2018) Improving cnn-rnn hybrid networks for handwriting recognition. In: 2018 16th international conference on frontiers in handwriting recognition (ICFHR), pp 80–85

  18. de Oliveira LES, Sabourin R, Bortolozzi F, Suen CY (2002) Automatic recognition of handwritten numerical strings: a recognition and verification strategy. IEEE Trans Pattern Anal Mach Intell 24:1438–1454

    Article  Google Scholar 

  19. Elleuch M, Maalej R, Kherallah M (2016) A new design based-svm of the cnn classifier architecture with dropout for offline arabic handwritten recognition. In: ICCS

  20. Eraqi HM, Abdelazeem S, Rashwan MAA (2016) Combining analytical and holistic strategies for handwriting recognition. In: 2016 15th IEEE international conference on machine learning and applications (ICMLA), pp 993–997

  21. Feng S (2019) A novel variational model for noise robust document image binarization. Neurocomputing 325:288–302

    Article  Google Scholar 

  22. Graves A (2013) Generating sequences with recurrent neural networks. arXiv:1308.0850

  23. Grdiet P, Garg NK (2013) Binarization techniques used for grey scale images

  24. Hallale SB, Salunke PGD (2013) Twelve directional feature extraction for handwritten english character recognition

  25. He S, Schomaker L (2019) Deep adaptive learning for writer identification based on single handwritten word images. Pattern Recogn 88:64–74

    Article  Google Scholar 

  26. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9:1735–1780

    Article  Google Scholar 

  27. Hu M-K (1962) Visual pattern recognition by moment invariants. IRE Trans Information Theory 8:179–187

    MATH  Google Scholar 

  28. Hussein ME, Torki M, Elsallamy A, Fayyaz M (2014) Alexu-word: a new dataset for isolated-word closed-vocabulary offline arabic handwriting recognition. arXiv:1411.4670

  29. Ingle RR, Fujii Y, Deselaers T, Baccash J, Popat AC A scalable handwritten text recognition system. arXiv:1904.09150

  30. Jaderberg M, Simonyan K, Vedaldi A, Zisserman A (2015) Reading text in the wild with convolutional neural networks. Int J Comput Vis 116:1–20

    Article  MathSciNet  Google Scholar 

  31. Jaderberg M, Vedaldi A, Zisserman A (2014) Deep features for text spotting. In: ECCV

  32. Jayech K, Mahjoub MA, Amara NEB (2016) Arabic handwritten word recognition based on dynamic bayesian network. Int Arab J Inf Technol 13:1024–1031

    Google Scholar 

  33. Kacalak W, Stuart KD, Majewski M (2007) Selected problems of intelligent handwriting recognition. In: Analysis and design of intelligent systems using soft computing techniques

  34. Kadhm MS, Karim A, Hassan A (2015) Handwriting word recognition based on svm classifier

  35. Kessentini Y, Paquet T, Hamadou AB (2010) Off-line handwritten word recognition using multi-stream hidden markov models. Pattern Recogn Lett 31:60–70

    Article  Google Scholar 

  36. Kleber F, Fiel S, Diem M, Sablatnig H (2013) Cvl-database: an off-line database for writer retrieval, writer identification and word spotting. In: 2013 12th international conference on document analysis and recognition, pp 560–564

  37. Kozielski M, Forster J, Ney H (2012) Moment-based image normalization for handwritten text recognition. In: 2012 international conference on frontiers in handwriting recognition, pp 256–261

  38. Lawgali A (2015) A survey on arabic character recognition

  39. LeCun Y (1998) Gradient-based learning applied to document recognition

  40. Maalej R, Kherallah M (2018) Convolutional neural network and blstm for offline arabic handwriting recognition. In: 2018 international arab conference on information technology (ACIT), pp 1–6

  41. Mamathah HR, Madireddi S, Srikantamurthy KR (2012) Performance analysis of various filters for de-noising of handwritten kannada documents

  42. Marti U-V, Bunke H (2002) The iam-database: an english sentence database for offline handwriting recognition. Int J Doc Anal Recognit 5:39–46

    Article  Google Scholar 

  43. Mhiri M, Desrosiers C, Cheriet M (2018) Convolutional pyramid of bidirectional character sequences for the recognition of handwritten words. Pattern Recogn Lett 111:87–93

    Article  Google Scholar 

  44. Mhiri M, Desrosiers C, Cheriet M (2019) Word spotting and recognition via a joint deep embedding of image and text. Pattern Recogn 88:312–320

    Article  Google Scholar 

  45. Panwar S, Nain N (2012) Handwritten text documents binarization and skew normalization approaches. In: 2012 4th international conference on intelligent human computer interaction (IHCI), pp 1–6

  46. Pechwitz M, Maddouri S, Märgner V, Ellouze N, Amiri H (2002) Ifn/enit - database of handwritten arabic words

  47. Ptucha RW, Such FP, Pillai S, Brockler F, Singh V, Paul H (2019) Intelligent character recognition using fully convolutional neural networks. Pattern Recogn 88:604–613

    Article  Google Scholar 

  48. Rajora S, Kumar Vishwakarma D, Singh K, Prasad M (2018) Csgi: a deep learning based approach for marijuana leaves strain classification. In: 2018 IEEE 9th annual information technology, electronics and mobile communication conference (IEMCON). IEEE, pp 209–214

  49. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: MICCAI

  50. Sharan A (1993) Character recognition using fourier coefficients, Ph.D. thesis

  51. Shi B, Bai X, Yao C (2017) An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Trans Pattern Anal Mach Intell 39:2298–2304

    Article  Google Scholar 

  52. Shi X, Chen Z, Wang H, Yeung D-Y, Wong W-K, chun Woo W (2015) Convolutional lstm network: a machine learning approach for precipitation nowcasting. In: NIPS

  53. Sonkusare M, Sahu N (2016) A survey on handwritten character recognition (hcr) techniques for english alphabets. Adv Vis Comput Int J 3(1):1–12

    Article  Google Scholar 

  54. Sudarsan D, Joseph S (2018) A novel approach for handwriting recognition in malayalam manuscripts using contour detection and convolutional neural nets. In: 2018 international conference on advances in computing, communications and informatics (ICACCI), pp 1818–1824

  55. Sueiras J, Ruíz V, Sánchez Á, Vélez JF (2018) Offline continuous handwriting recognition using sequence to sequence neural networks. Neurocomputing 289:119–128

    Article  Google Scholar 

  56. Sulaiman WNA, Omar MA (2019) Nasrudin, degraded historical document binarization: a review on issues, challenges, techniques and future directions

  57. Sulaiman A, Omar K, Nasrudin MF, Arram A (2019) Length independent writer identification based on the fusion of deep and hand-crafted descriptors. IEEE Access 7:91772–91784

    Article  Google Scholar 

  58. Tavoli R, Keyvanpour MR, Mozaffari S (2018) Statistical geometric components of straight lines (sgcsl) feature extraction method for offline arabic/persian handwritten words recognition. IET Image Process 12:1606–1616

    Article  Google Scholar 

  59. Wang W-L, Tang MH (2015) A normalization process to standardize handwriting data collected from multiple resources for recognition. Procedia Computer Science 61:402–409. Complex Adaptive Systems San Jose, CA November 2–4, 2015

    Article  Google Scholar 

  60. Xue H, Govindaraju V (2006) Hidden markov models combining discrete symbols and continuous attributes in handwriting recognition. IEEE Trans Pattern Anal Mach Intell 28:458–462

    Article  Google Scholar 

  61. Yan H (1993) Skew correction of document images using interline cross-correlation. CVGIP: Graphical Model and Image Processing 55:538–543

    Google Scholar 

Download references

Acknowledgements

This research was funded by the Ministry of Higher Education in Malaysia by the grant number FRGS/1/2016/ICT02/UKM/01/1.

Author information

Authors and Affiliations

  1. Pattern Recognition Research Group, Center of Artificial Intelligence Technology, Faculty of Information Science and Technology University Kebangsaan Malaysia (UKM), Bangi, Selangor, 43600, Malaysia

    Alaa Sulaiman, Khairuddin Omar & Mohammad F. Nasrudin

Authors
  1. Alaa Sulaiman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Khairuddin Omar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohammad F. Nasrudin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alaa Sulaiman.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sulaiman, A., Omar, K. & Nasrudin, M.F. Two streams deep neural network for handwriting word recognition. Multimed Tools Appl 80, 5473–5494 (2021). https://doi.org/10.1007/s11042-020-09923-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09923-1

Keywords

Search

Navigation