Attention Enhanced ConvNet-RNN for Chinese Vehicle License Plate Recognition

  • Shiming Duan
  • Wei Hu
  • Ruirui LiEmail author
  • Wei Li
  • Shihao Sun
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11257)


As an important part of intelligent transportation system, vehicle license plate recognition requires high accuracy in an open environment. While a lot of approaches have been proposed, and achieved good performance to some extent, these approaches still have problems, for example, in the condition of characters’ distortion or partial occlusion. Segmentation-free VLPR systems compute the label in one pass using Long Short-Term Memory Network (LSTM), without individual segmentation step, their results tend to be not influenced by the segmentation accuracy. Based on the idea of Segmentation-free VLPR, this paper proposed an attention enhanced ConvNet-RNN (AC-RNN) for accurate Chinese Vehicle License Plate Recognition. The attention mechanism helps to locate the important instances in the step of recognition. While the ConvNet is used to extract features, the recurrent neural networks (RNN) with connectionist temporal classification (CTC) are applied for sequence labeling. The proposed AC-RNN was trained on a large generated dataset which contains various types of license plates in China. The AC-RNN could figure out the vehicle license even in cases of light changing, spatial distortion and partial blurry. Experiments showed that the AC-RNN performs better on the testing real images, increasing about 5% on accuracy, compared with classic ConvNet-RNN [8].


Vehicle license plate recognition Recurrent neural networks Long Short-Term Memory Network Attention 


  1. 1.
    Aboura, K., Al-Hmouz, R.: An overview of image analysis algorithms for license plate recognition. Organizacija 50(3), 285–295 (2017)CrossRefGoogle Scholar
  2. 2.
    Abtahi, F., Zhu, Z., Burry, A.M.: A deep reinforcement learning approach to character segmentation of license plate images. In: 2015 14th IAPR International Conference on Machine Vision Applications (MVA), pp. 539–542, May 2015.
  3. 3.
    Angara, N.S.S.: Automatic license plate recognition using deep learning techniques (2015)Google Scholar
  4. 4.
    Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
  5. 5.
    Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., Bengio, Y.: End-to-end attention-based large vocabulary speech recognition. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4945–4949. IEEE (2016)Google Scholar
  6. 6.
    Bhunia, A.K., Konwer, A., Bhowmick, A., Bhunia, A.K., Roy, P.P., Pal, U.: Script identification in natural scene image and video frame using attention based convolutional-LSTM network (2018)Google Scholar
  7. 7.
    Calvo-Zaragoza, J., Valero-Mas, J.J., Pertusa, A.: End-to-end optical music recognition using neural networks. In: Proceedings of the 18th International Society for Music Information Retrieval Conference, Suzhou, China, pp. 23–27 (2017)Google Scholar
  8. 8.
    Cheang, T.K., Chong, Y.S., Yong, H.T.: Segmentation-free vehicle license plate recognition using ConvNet-RNN (2017)Google Scholar
  9. 9.
    Chorowski, J.K., Bahdanau, D., Serdyuk, D., Cho, K., Bengio, Y.: Attention-based models for speech recognition. In: Advances in Neural Information Processing Systems, pp. 577–585 (2015)Google Scholar
  10. 10.
    Ding, H., et al.: A compact CNN-DBLSTM based character model for offline handwriting recognition with Tucker decomposition. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 01, pp. 507–512, November 2017.
  11. 11.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)Google Scholar
  12. 12.
    Gou, C., Wang, K., Yao, Y., Li, Z.: Vehicle license plate recognition based on extremal regions and restricted Boltzmann machines. IEEE Trans. Intell. Transp. Syst. 17(4), 1096–1107 (2016)CrossRefGoogle Scholar
  13. 13.
    He, S., Yang, C., Pan, J.S.: The research of chinese license plates recognition based on CNN and length\_feature. In: Fujita, H., Ali, M., Selamat, A., Sasaki, J., Kurematsu, M. (eds.) Trends in Applied Knowledge-Based Systems and Data Science, pp. 389–397. Springer, Cham (2016). Scholar
  14. 14.
    Hui Li, C.S.: Reading car license plates using deep convolutional neural networks and LSTMS (2016)Google Scholar
  15. 15.
    Hurtik, P., Vajgl, M.: Automatic license plate recognition in difficult conditions - technical report. In: Fuzzy Systems Association and International Conference on Soft Computing and Intelligent Systems (2017)Google Scholar
  16. 16.
    Laroca, R., et al.: A robust real-time automatic license plate recognition based on the YOLO detector. In: International Joint Conference on Neural Networks (2018)Google Scholar
  17. 17.
    Lee, C.Y., Osindero, S.: Recursive recurrent nets with attention modeling for OCR in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2231–2239 (2016)Google Scholar
  18. 18.
    Li, H., Wang, P., Shen, C.: Towards end-to-end car license plates detection and recognition with deep neural networks (2017)Google Scholar
  19. 19.
    Li, H., Wang, P., You, M., Shen, C.: Reading car license plates using deep neural networks. Image Vis. Comput. 72, 14–23 (2018)., Scholar
  20. 20.
    Liu, W., et al.: SSD: single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). Scholar
  21. 21.
    Liu, Y., Huang, H.: Car plate character recognition using a convolutional neural network with shared hidden layers. In: 2015 Chinese Automation Congress (CAC), pp. 638–643, November 2015.
  22. 22.
    Luong, M.T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 (2015)
  23. 23.
    Masood, S.Z., Shu, G., Dehghan, A., Ortiz, E.G.: License plate detection and recognition using deeply learned convolutional neural networks (2017)Google Scholar
  24. 24.
    Mnih, V., Heess, N., Graves, A., et al.: Recurrent models of visual attention. In: Advances in Neural Information Processing Systems, pp. 2204–2212 (2014)Google Scholar
  25. 25.
    Mubarak, H., Ibrahim, A.O., Elwasila, A., Bushra, S., Ahmed, A.: A framework for automatic license number plate recognition in sudanese vehicles (2017)Google Scholar
  26. 26.
    Pant, A.K., Gyawali, P.K., Acharya, S.: Automatic nepali number plate recognition with support vector machines. In: International Conference on Software, Knowledge, Information Management and Applications (2015)Google Scholar
  27. 27.
    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788 (2016)Google Scholar
  28. 28.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)Google Scholar
  29. 29.
    Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(11), 2298–2304 (2017). Scholar
  30. 30.
    Tian, Z., Huang, W., He, T., He, P., Qiao, Y.: Detecting text in natural image with connectionist text proposal network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 56–72. Springer, Cham (2016). Scholar
  31. 31.
    Wang, J., Bacic, B., Yan, W.Q.: An effective method for plate number recognition. Multimed. Tools Appl. 77(2), 1679–1692 (2018). Scholar
  32. 32.
    Wel, E.V.D., Ullrich, K.: Optical music recognition with convolutional sequence-to-sequence models (2017)Google Scholar
  33. 33.
    Zang, D.: Vehicle license plate recognition using visual attention model and deep learning. J. Electron. Imaging 24(3), 033001 (2015)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Shiming Duan
    • 1
  • Wei Hu
    • 1
  • Ruirui Li
    • 1
    Email author
  • Wei Li
    • 1
  • Shihao Sun
    • 1
  1. 1.Beijing University of Chemical TechnologyBeijingChina

Personalised recommendations