Skip to main content
SpringerLink
Log in

Towards end-to-end car license plate location and recognition in unconstrained scenarios

  • S.I. : NCACVIP
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Benefiting from the rapid development of convolutional neural networks, the performance of car license plate detection and recognition has been largely improved. Nonetheless, most existing methods solve detection and recognition problems separately, and focus on specific scenarios, which hinders the deployment for real-world applications. To overcome these challenges, we present an efficient and accurate framework to solve the license plate detection and recognition tasks simultaneously. It is a lightweight and unified deep neural network, that can be optimized end-to-end and work in real-time. Specifically, for unconstrained scenarios, an anchor-free method is adopted to efficiently detect the bounding box and four corners of a license plate, which are used to extract and rectify the target region features. Then, a novel convolutional neural network branch is designed to further extract features of characters without segmentation. Finally, the recognition task is treated as sequence labeling problems, which are solved by Connectionist Temporal Classification (CTC) directly. Several public datasets including images collected from different scenarios under various conditions are chosen for evaluation. Experimental results indicate that the proposed method significantly outperforms the previous state-of-the-art methods in both speed and precision.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Du S, Ibrahim M, Shehata M, Badawy W (2013) Automatic license plate recognition (ALPR): a state-of-the-art review. IEEE Trans Circuits Syst Video Technol 23(2):311–325

    Article  Google Scholar 

  2. Silva SM, Jung CR (2018) License plate detection and recognition in unconstrained scenarios. In: Proceedings of the European Conference Computer Vision (ECCV)

  3. Björklund T, Fiandrotti A, Annarumma M, Francini G, Magli E (2019) Robust license plate recognition using neural networks trained on synthetic images. Pattern Recognit 93:134–146

    Article  Google Scholar 

  4. Laroca R, Zanlorensi LA, Gonçalves GR, Todt E, Schwartz WR, Menotti D (2019) An efficient and layout-independent automatic license plate recognition system based on the yolo detector. arXiv:1909.01754

  5. He MX, Hao P (2020) Robust automatic recognition of Chinese license plates in natural scenes. IEEE Access 8:173804–173814. https://doi.org/10.1109/ACCESS.2020.3026181

    Article  Google Scholar 

  6. Silva SM, Jung CR (2020) Real-time license plate detection and recognition using deep convolutional neural networks. J Visual Commun Image Represent 71:102773

    Article  Google Scholar 

  7. Weihong W, Jiaoyang T (2020) Research on license plate recognition algorithms based on deep learning in complex environment. IEEE Access 8:91661–91675. https://doi.org/10.1109/ACCESS.2020.2994287

    Article  Google Scholar 

  8. Yuan Y, Zou W, Zhao Y, Wang X, Hu X, Komodakis N (2017) A robust and efficient approach to license plate detection. IEEE Trans Image Process 26(3):1102–1114

    Article  MathSciNet  MATH  Google Scholar 

  9. Xie L, Ahmad T, Jin L, Liu Y, Zhang S (2018) A new CNN-based method for multi-directional car license plate detection. IEEE Trans Intell Transp Syst 19(2):507–517

    Article  Google Scholar 

  10. Zhuang J, Hou S, Wang Z, Zha ZJ (2018) Towards human-level license plate recognition. In: Proceedings of the European Conference on Computer Vision (ECCV)

  11. Liu C, Chang F (2019) Hybrid cascade structure for license plate detection in large visual surveillance scenes. IEEE Trans Intell Transp Syst 20(6):2122–2135

    Article  Google Scholar 

  12. Al-Shemarry MS, Li Y, Abdulla S (2020) An efficient texture descriptor for the detection of license plates from vehicle images in difficult conditions. IEEE Trans Intell Transp Syst 21(2):553–564

    Article  Google Scholar 

  13. Redmon J, Farhadi A (2018) Yolov3: an incremental improvement. arXiv:1804.02767

  14. Wei L, Dragomir A, Dumitru E, Christian S, Scott R, Chengyang F, Alexander B (2016) SSD: Single shot multibox detector. In: Proceedings of the European Conference on Computer Vision (ECCV)

  15. Ren S, He K, Girshick R, Sun J (2017) Faster r-cnn: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149

    Article  Google Scholar 

  16. Wang J, Huang H, Qian X, Cao J, Dai Y (2018) Sequence recognition of Chinese license plates. Neurocomputing 317:149–158

    Article  Google Scholar 

  17. Li H, Wang P, Shen C (2019) Toward end-to-end car license plate detection and recognition with deep neural networks. IEEE Trans Intell Transp Syst 20(3):1126–1136

    Article  Google Scholar 

  18. Xu Z, Yang W, Meng A, Lu N, Huang H, Ying C, Huang L (2018) Towards end-to-end license plate detection and recognition: a large dataset and baseline. In: Proceedings of the European Conference on Computer Vision (ECCV)

  19. Zhao Z, Zheng P, Xu S, Wu X (2019) Object detection with deep learning: a review. IEEE Trans Neural Netw Learn Syst 30(11):3212–3232

    Article  Google Scholar 

  20. Hsu G, Chen J, Chung Y (2013) Application-oriented license plate recognition. IEEE Trans Veh Technol 62(2):552–561

    Article  Google Scholar 

  21. Ashtari AH, Nordin MJ, Fathy M (2014) An Iranian license plate recognition system based on color features. IEEE Trans Intell Transp Syst 15(4):1690–1705

    Article  Google Scholar 

  22. Yu S, Li B, Zhang Q, Liu C, Meng MQH (2015) A novel license plate location method based on wavelet transform and EMD analysis. Pattern Recognit 48(1):114–125

    Article  Google Scholar 

  23. Setumin S, Sheikh UU, Abu-Bakar SAR (2010) Character-based car plate detection and localization. In: Proceedings of the International Conference on Information Science, Signal Processing and their Applications (ISSPA 2010), pp 737–740

  24. Li B, Tian B, Li Y, Wen D (2013) Component-based license plate detection using conditional random field model. IEEE Trans Intell Transp Syst 14(4):1690–1699

    Article  Google Scholar 

  25. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  26. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision Pattern Reorganization (CVPR), pp 779–788

  27. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision Pattern Recognition (CVPR)

  28. Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision Pattern Recognition (CVPR)

  29. Laroca R, Severo E, Zanlorensi LA, Oliveira LS, Gonçalves GR, Schwartz WR, Menotti D (2018) A robust real-time automatic license plate recognition based on the yolo detector. In: Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN), pp 1–10

  30. Law H, Deng J (2018) Cornernet: detecting objects as paired keypoints. In: Proceedings of the European Conference on Computer Vision (ECCV)

  31. Zhou X, Wang D, Krähenbühl P (2019) Objects as points. arXiv:1904.07850

  32. Zhang J, Li Y, Li T, Xun L, Shan C (2019) License plate localization in unconstrained scenes using a two-stage CNN–RNN. IEEE Sens J 19(13):5256–5265

    Article  Google Scholar 

  33. Wu HHP, Chen HH, Wu RJ, Shen DF (2006) License plate extraction in low resolution video. In: Proceedings of the IEEE Conference on Computer Vision Pattern Recognition (CVPR), vol 1, pp 824–827

  34. Goel S, Dabas S (2013) Vehicle registration plate recognition system using template matching. In: Proceedings of the International Conference on Signal Processing Communication (ICSC), pp 315–318

  35. Gou C, Wang K, Yao Y, Li Z (2016) Vehicle license plate recognition based on extremal regions and restricted Boltzmann machines. IEEE Trans Intell Transp Syst 17(4):1096–1107

    Article  Google Scholar 

  36. Hendry Chen RC (2019) Automatic license plate recognition via sliding-window darknet-yolo deep learning. Image Vis Comput 87:47–56

    Article  Google Scholar 

  37. Khan MA, Sharif M, Javed MY, Akram T, Yasmin M, Saba T (2018) License number plate recognition system using entropy-based features selection approach with SVM. IET Image Proc 12(2):200–209

    Article  Google Scholar 

  38. Bulan O, Kozitsky V, Ramesh P, Shreve M (2017) Segmentation- and annotation-free license plate recognition with deep localization and failure identification. IEEE Trans Intell Transp Syst 18(9):2351–2363

    Article  Google Scholar 

  39. Henry C, Ahn SY, Lee S (2020) Multinational license plate recognition using generalized character sequence detection. IEEE Access 8:35185–35199. https://doi.org/10.1109/ACCESS.2020.2974973

    Article  Google Scholar 

  40. Selmi Z, Halima MB, Pal U, Alimi MA (2020) Delp-dar system for license plate detection and recognition. Pattern Recognit Lett 129:213–223

    Article  Google Scholar 

  41. He K, Gkioxari G, Dollar P, Girshick R (2017) Mask r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV)

  42. Zherzdev S, Gruzdev A (2018) Lprnet: License plate recognition via deep neural networks. arXiv:1806.10447

  43. Yang Y, Li D, Duan Z (2018) Chinese vehicle license plate recognition using kernel-based extreme learning machine with deep convolutional features. IET Intel Transport Syst 12(3):213–219

    Article  Google Scholar 

  44. Resende Gonçalves G, Alves Diniz M, Laroca R, Menotti D, Robson Schwartz W (2018) Real-time automatic license plate recognition through deep multi-task networks. In: Proceeding of SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pp 110–117, https://doi.org/10.1109/SIBGRAPI.2018.00021

  45. Yue W, Jianmin L (2017) License plate recognition using deep FCN. In: Proceedings of the International Conference on Cognitive System Signal Processing (ICCSIP)

  46. Alex G, Santiago F, Faustino G, JüRgen S (2006) Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the International Conference on Machine Learning (ICML)

  47. Li H, Wang P, You M, Shen C (2018) Reading car license plates using deep neural networks. Image Vision Comput 72:14–23

    Article  Google Scholar 

  48. Liu Y, Huang H, Cao J, Huang T (2018) Convolutional neural networks-based intelligent recognition of Chinese license plates. Soft Comput 22:2403–2419

    Article  Google Scholar 

  49. Zhang C, Wang Q, Li X (2020) Eq-lpr: efficient quality-aware license plate recognition. In: Proceedings of the IEEE International Conference on Image Processing (ICIP), pp 653–657

  50. Shi B, Bai X, Yao C (2017) An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Trans Pattern Anal Mach Intell 39(11):2298–2304

    Article  Google Scholar 

  51. Li H, Wang P, Shen C (2017) Towards end-to-end text spotting with convolutional recurrent neural networks. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV)

  52. He T, Tian Z, Huang W, Shen C, Qiao Y, Sun C (2018) An end-to-end textspotter with explicit alignment and attention. In: Proceedings of the IEEE Conference on Computer Vision Pattern Recognition (CVPR), pp 5020–5029

  53. Wang W, Xie E, Song X, Zang Y, Wang W, Lu T, Yu G, Shen C (2019) Efficient and accurate arbitrary-shaped text detection with pixel aggregation network. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV)

  54. Lin TY, Dollar P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision Pattern Recognition (CVPR)

  55. Wang W, Xie E, Li X, Hou W, Lu T, Yu G, Shao S (2019) Shape robust text detection with progressive scale expansion network. In: Proceedings of the IEEE Conferences on Computer Vision Pattern Recognition (CVPR)

  56. Lin TY, Goyal P, Girshick R, He K, Dollar P (2017) Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV)

  57. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980

  58. Deng J, Dong W, Socher R, Li L, Kai L, Li FF (2009) Imagenet: a large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision Pattern Recognition (CVPR), pp 248–255

  59. Špaňhel J, Sochor J, Juránek R, Herout A, Maršík L, Zemčík P (2017) Holistic recognition of low quality license plates by cnn using track annotated data. In: Proceedings of the IEEE International Conference on Advanced Video Signal Based Surveillance (AVSS), pp 1–6

Download references

Author information

Authors and Affiliations

  1. the Department of Research and Development, AInnovation Technology Co., Ltd., No.6 Xingzhi Road, Qixia District, Nanjing, China

    Shuxin Qin

  2. Nanjing University of Posts and Telecommunications, No.9 Wenyuan Road, Qixia District, Nanjing, China

    Sijiang Liu

Authors
  1. Shuxin Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sijiang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shuxin Qin.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest to this work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qin, S., Liu, S. Towards end-to-end car license plate location and recognition in unconstrained scenarios. Neural Comput & Applic 34, 21551–21566 (2022). https://doi.org/10.1007/s00521-021-06147-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06147-8

Keywords

Search

Navigation