Skip to main content
SpringerLink
Log in

How Far Deep Learning Systems for Text Detection and Recognition in Natural Scenes are Affected by Occlusion?

  • Conference paper
  • First Online:
Document Analysis and Recognition – ICDAR 2021 Workshops (ICDAR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12916))

Included in the following conference series:

Abstract

With the rise of deep learning, significant advances in scene text detection and recognition in natural images have been made. However, the severe impact threat to the algorithm’s performance caused by occlusion still consists of an open issue due to the lack of consistent real-world datasets, richer annotations, and evaluations in the specific occlusion problem. Therefore, unlike previous works in this field, our paper addresses occlusions in scene text recognition. The goal is to evaluate the effectiveness and efficiency of existing deep architectures for scene text detection and recognition in various occlusion levels. First, we investigated state-of-the-art scene text and recognition methods and evaluated these current deep architectures performances on ICDAR 2015 dataset without any generated occlusion. Second, we created a methodology to generate large datasets of scene text in natural images with ranges of occlusion between 0 and 100%. From this methodology, we produced the ISTD-OC, a dataset derivated from the ICDAR 2015 database to evaluate deep architectures under different levels of occlusion. The results demonstrated that these existing deep architectures that have achieved state-of-the-art are still far from understanding text instances in a real-world scenario. Unlike the human vision systems, which can comprehend occluded instances by contextual reasoning and association, our extensive experimental evaluations show that current scene text recognition models are inefficient when high occlusions exist in a scene. Nevertheless, for scene text detection, segmentation-based methods, such as PSENet and PAN, are more robust in predicting higher levels of occluded texts. In contrast, methods that detect at the character level, such as CRAFT, are unsatisfactory to heavy occlusions. When it comes to recognition, attention-based methods that benefit contextual information have performed better than CTC-based methods.

Supported by Foundation for the Support of Science and Technology of the State of Pernambuco (FACEPE), Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001, and CNPq - Brazilian agencies.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/alinesoares1/ISTD-OC-Dataset.

References

  1. Adak, C., Chaudhuri, B.B., Blumenstein, M.: Impact of struck-out text on writer identification. In: 2017 International Joint Conference on Neural Networks (IJCNN), pp. 1465–1471. IEEE (2017)

    Google Scholar 

  2. Baek, J., Han, Y., Kim, J.O., Lee, J., Park, S.: What is wrong with scene text recognition model comparisons? Dataset and model analysis (2019)

    Google Scholar 

  3. Baek, Y., Lee, B., Han, D., Yun, S., Lee, H.: Character region awareness for text detection (2019)

    Google Scholar 

  4. Borisyuk, F., Gordo, A., Sivakumar, V.: Rosetta: large scale system for text detection and recognition in images. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 71–79 (2018)

    Google Scholar 

  5. Chen, X., Jin, L., Zhu, Y., Luo, C., Wang, T.: Text recognition in the wild: a survey. ACM Comput. Surv. (CSUR) 54, 1–35 (2020)

    Google Scholar 

  6. Efimova, V., Shalamov, V., Filchenkov, A.: Synthetic dataset generation for text recognition with generative adversarial networks, vol. 1143315, p. 62 (2020). https://doi.org/10.1117/12.2558271

  7. Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 369–376 (2006)

    Google Scholar 

  8. Gupta, A., Vedaldi, A., Zisserman, A.: Synthetic data for text localisation in natural images. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, December 2016, pp. 2315–2324 (2016). https://doi.org/10.1109/CVPR.2016.254

  9. Jaderberg, M., Simonyan, K., Vedaldi, A., Zisserman, A.: Synthetic data and artificial neural networks for natural scene text recognition. arXiv preprint arXiv:1406.2227 (2014)

  10. Karatzas, D., et al.: ICDAR 2015 competition on robust reading. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pp. 1156–1160. IEEE (2015)

    Google Scholar 

  11. Lin, H., Yang, P., Zhang, F.: Review of scene text detection and recognition. Arch. Comput. Methods Eng. 27(2), 433–454 (2020). https://doi.org/10.1007/s11831-019-09315-1

    Article  Google Scholar 

  12. Liu, W., Chen, C., Wong, K.Y.K., Su, Z., Han, J.: Star-net: a spatial attention residue network for scene text recognition. In: BMVC, vol. 2, p. 7 (2016)

    Google Scholar 

  13. Long, S., He, X., Yao, C.: Scene text detection and recognition: the deep learning era. Int. J. Comput. Vis. 129, 1–24 (2020)

    Google Scholar 

  14. Mishra, A., Alahari, K., Jawahar, C.: Scene text recognition using higher order language priors (2012)

    Google Scholar 

  15. Mittal, A., Shivakumara, P., Pal, U., Lu, T., Blumenstein, M., Lopresti, D.: A new context-based method for restoring occluded text in natural scene images. In: Bai, X., Karatzas, D., Lopresti, D. (eds.) DAS 2020. LNCS, vol. 12116, pp. 466–480. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-57058-3_33

    Chapter  Google Scholar 

  16. Neto, A.F.D.S., Bezerra, B.L.D., Toselli, A.H.: Towards the natural language processing as spelling correction for offline handwritten text recognition systems. Appl. Sci. 10(21), 7711 (2020). https://doi.org/10.3390/app10217711

    Article  Google Scholar 

  17. Nisa, H., Thom, J.A., Ciesielski, V., Tennakoon, R.: A deep learning approach to handwritten text recognition in the presence of struck-out text. In: International Conference Image and Vision Computing New Zealand (December 2019). https://doi.org/10.1109/IVCNZ48456.2019.8961024

  18. Qi, J., et al.: Occluded video instance segmentation. arXiv preprint arXiv:2102.01558 (2021)

  19. Raisi, Z., Naiel, M.A., Fieguth, P., Jun, C.V.: Text detection and recognition in the wild: a review, pp. 13–15 (2020)

    Google Scholar 

  20. Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(11), 2298–2304 (2016)

    Article  Google Scholar 

  21. Shi, B., Wang, X., Lyu, P., Yao, C., Bai, X.: Robust scene text recognition with automatic rectification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4168–4176 (2016)

    Google Scholar 

  22. Veit, A., Matera, T., Neumann, L., Matas, J., Belongie, S.: Coco-text: dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140 (2016)

  23. Wang, J., Hu, X.: Gated recurrent convolution neural network for OCR. In: Advances in Neural Information Processing Systems, vol. 30, pp. 335–344 (2017)

    Google Scholar 

  24. Wang, K., Babenko, B., Belongie, S.: End-to-end scene text recognition. In: 2011 International Conference on Computer Vision, pp. 1457–1464. IEEE (2011)

    Google Scholar 

  25. Wang, W., et al.: Shape robust text detection with progressive scale expansion network. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2019, pp. 9328–9337 (2019). https://doi.org/10.1109/CVPR.2019.00956

  26. Wang, W., et al.: Efficient and accurate arbitrary-shaped text detection with pixel aggregation network. In: Proceedings of the IEEE International Conference on Computer Vision, October 2019, pp. 8439–8448 (2019). https://doi.org/10.1109/ICCV.2019.00853

  27. Yao, C., Bai, X., Liu, W., Ma, Y., Tu, Z.: Detecting texts of arbitrary orientations in natural images. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1083–1090. IEEE (2012)

    Google Scholar 

  28. Yuan, T.L., Zhu, Z., Xu, K., Li, C.J., Hu, S.M.: Chinese text in the wild. arXiv preprint arXiv:1803.00085 (2018)

  29. Zhao, F., Feng, J., Zhao, J., Yang, W., Yan, S.: Robust LSTM-autoencoders for face de-occlusion in the wild. IEEE Trans. Image Process. 27, 778–790 (2016)

    Article  MathSciNet  Google Scholar 

  30. Zhou, X., et al.: EAST: an efficient and accurate scene text detector (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Pernambuco, Recife, Brazil

    Aline Geovanna Soares, Byron Leite Dantas Bezerra & Estanislau Baptista Lima

Authors
  1. Aline Geovanna Soares
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Byron Leite Dantas Bezerra
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Estanislau Baptista Lima
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Byron Leite Dantas Bezerra .

Editor information

Editors and Affiliations

  1. Boise State University, Boise, ID, USA

    Elisa H. Barney Smith

  2. Indian Statistical Institute, Kolkata, India

    Umapada Pal

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Geovanna Soares, A., Leite Dantas Bezerra, B., Baptista Lima, E. (2021). How Far Deep Learning Systems for Text Detection and Recognition in Natural Scenes are Affected by Occlusion?. In: Barney Smith, E.H., Pal, U. (eds) Document Analysis and Recognition – ICDAR 2021 Workshops. ICDAR 2021. Lecture Notes in Computer Science(), vol 12916. Springer, Cham. https://doi.org/10.1007/978-3-030-86198-8_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-86198-8_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-86197-1

  • Online ISBN: 978-3-030-86198-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation