Skip to main content
SpringerLink
Log in

LMix: regularization strategy for convolutional neural networks

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Deep convolutional neural networks perform well in the field of computer vision, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. Therefore, proper regularization strategies are needed to alleviate these problems. Currently, regularization strategies with mixed sample data augmentation perform very well, and these algorithms allow the network to generalize better, improve the baseline performance of the model. However, interpolation-based mixed sample data augmentation distorts the data distribution, while masking-based mixed sample data augmentation results in excessive information loss for overly regular shapes of masks. Although mixed sample data augmentation is proven to be an effective method to improve the baseline performance, generalization ability and robustness of deep convolutional models, there is still room for improvement in terms of maintaining the of image local consistency and image data distribution. In this paper, we propose a new mixed sample data augmentation-LMix, which uses random masking to increase the number of masks in the image to maintain the data distribution, and high-frequency filtering to sharpen the image to highlight recognition regions. We applied the method to train CIFAR-10, CIFAR-100, SVHN, and Tiny-ImageNet datasets under the PreAct-ResNet18 model to evaluate the method, and obtained the latest results of 96.32, 79.85, 97.01, and 64.16%, respectively, which are 1.70, 4.73, and 8.06% higher than the optimal baseline accuracy. The LMix algorithm improves the generalization ability of the state-of-the-art neural network architecture and enhances the robustness to adversarial samples.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Availability of data and materials

All of our data sets come from public data sets.You can go to the corresponding official website to download.

References

  1. P. Foret, A. Kleiner, H. Mobahi, B. Neyshabur, Sharpness-aware minimization for efficiently improving generalization. (2021), arXiv:2010.01412

  2. D.K. Mahajan, R.B. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, L.V. Maaten, Exploring the limits of weakly supervised pretraining. (2018), arXiv:1805.00932

  3. M. Tan, Q. Le, EfficientNetV2: smaller models and faster training. (2021), ArXiv:2104.00298

  4. R.G. Lopes, D. Yin, B. Poole, J. Gilmer, E.D. Cubuk, Improving Robustness without sacrificing accuracy with patch Gaussian augmentation. (2019), arXiv:1906.02611

  5. K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition. 2016 IEEE conference on computer vision and pattern recognition (CVPR), 770-778 (2016) arXiv:1512.03385

  6. T. Devries, G.W. Taylor, Improved regularization of convolutional neural networks with Cutout. (2017), arXiv:1708.04552

  7. L. Taylor, G.S. Nitschke, Improving deep learning using generic data augmentation. (2017), arXiv:Learning

  8. H. Zhang, M. Cissé, Y. Dauphin, D. Lopez-Paz, Mixup: beyond empirical risk minimization. (2018), arXiv:1710.09412

  9. S. Yun, D. Han, S. Oh, S. Chun, J. Choe, Y.J. Yoo, CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 6022-6031(2019) arXiv:1905.04899

  10. V. Verma, A. Lamb, C. Beckham, A. Najafi, I. Mitliagkas, D. Lopez-Paz, Y. Bengio, Manifold Mixup: Better Representations by Interpolating Hidden States. ICML. (2019) arXiv:1806.05236

  11. J. Kim, W. Choo, H.O. Song, Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup,(2020), arXiv:2009.06962

  12. E. Harris, A. Marcu, M. Painter, M. Niranjan, A. Prügel-Bennett, J.S. Hare, FMix: Enhancing Mixed Sample Data Augmentation. (2020) arXiv:2002.12047

  13. Chapelle, O., Weston, J., Bottou, L., apnik,V.: Vicinal risk minimization, in Advances in neural information processing systems. 416–422 (2001)

  14. Y. Yuan, X. Chen, J. Wang, Object-contextual representations for semantic segmentation. (2020), arXiv:1909.11065

  15. Dong, C., Loy, C.C., He, K., Tang, X.: Image Super-Resolution Using Deep Convolutional Networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 295–307 (2016)

    Article  Google Scholar 

  16. Vapnik, V.N.: The Nature of Statistic Learning Theory. (2000). https://doi.org/10.1007/978-1-4757-2440-0

    Article  Google Scholar 

  17. Krizhevsky, A. , Hinton, G.P.: Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases 1(4), (2009)

  18. Le, Y., Yang, X.S.: Tiny ImageNet visual recognition challenge. 529 CSN 231N 7, 3 (2015). https://tiny-imagenet.herokuapp.com

  19. K. He, X. Zhang, S. Ren, J. Sun, Identity mappings in deep residual networks. (2016), arXiv:1603.05027

  20. S. Zagoruyko, N. Komodakis, Wide residual networks. (2016), arXiv:1605.07146

  21. G. Huang, Z. Liu, K.Q. Weinberger, Densely connected convolutional networks. 2017 IEEE conference on computer vision and pattern recognition (CVPR), 2261-2269 (2017) arXiv:1608.06993

  22. D. Han, J. Kim, J. Kim, Deep pyramidal residual networks. 2017 IEEE conference on computer vision and pattern recognition (CVPR), 6307-6315 (2017) arXiv1610.02915

  23. I.J. Goodfellow, J. Shlens, C. Szegedy, Explaining and harnessing adversarial examples. (2015) CoRR, abs/1412.6572. arXiv:1412.6572

  24. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 84–90 (2012)

    Article  Google Scholar 

  25. Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)

    MATH  MathSciNet  Google Scholar 

  26. C. Zhang, S. Bengio, M. Hardt, B. Recht, O. Vinyals, Understanding deep learning requires rethinking generalization. (2017), arXiv:1611.03530

  27. Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning. J. Big Data 6, 1–48 (2019)

    Article  Google Scholar 

  28. H. Touvron, A. Vedaldi, M. Douze, H. J’egou, Fixing the train-test resolution discrepancy: FixEfficientNet. (2020), arXiv:2003.08237

  29. K. He, R.B. Girshick, P. Dollár, Rethinking ImageNet Pre-Training. 2019 IEEE/CVF international conference on computer vision (ICCV), 4917-4926 (2019) arXiv:1811.08883

  30. P. Sun, R. Zhang, Y. Jiang, T. Kong, C. Xu, W. Zhan, M. Tomizuka, L. Li, Z. Yuan, C. Wang, P. Luo, Sparse R-CNN: end-to-end object detection with learnable proposals. 2021 IEEE/CVF conference on computer vision and pattern recognition (CVPR), 14449-14458 (2021) arXiv:2011.12450

  31. S.H. Lee, S. Lee, B.C. Song, Vision transformer for small-size datasets. (2021), arXiv:2112.13492

  32. H. Bao, L. Dong, F. Wei, BEiT: BERT pre-training of image transformers. (2021), arXiv:2106.08254

  33. Fang, M., Chen, Z., Przystupa, K., Li, T., Majka, M., Kochan, O.: Examination of abnormal behavior detection based on improved YOLOv3. Electronics 10, 197 (2021)

    Article  Google Scholar 

  34. Song, W., Beshley, Przystupa, M., et al.: A software deep packet inspection system for network traffic analysis and anomaly detection. Sensors 20(6), 1637 (2020)

  35. Lu, X., Lu, X.: An efficient network for multi-scale and overlapped wildlife detection. SIViP (2022). https://doi.org/10.1007/s11760-022-02237-9

  36. Borkar, M., Cevher, V., McClellan, J.H.: Low computation and low latency algorithms for distributed sensor network initialization. SIViP 1, 133–148 (2007). https://doi.org/10.1007/s11760-007-0014-7

    Article  MATH  Google Scholar 

Download references

Funding

This work is funded by the National Natural Science Foundation of China under Grant No. 61772180, the Key R D plan of Hubei Province (2020BHB004, 2020BAB012).

Author information

Authors and Affiliations

  1. Hubei University of Technology, Wuhan, Hubei, China

    Linyu Yan, Kunpeng Zheng, Jinyao Xia & Ke Li

  2. Huazhong University of Science and Technology, Wuhan, Hubei, China

    Hefei Ling

Authors
  1. Linyu Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kunpeng Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinyao Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ke Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hefei Ling
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

LY and KZ completed the main manuscript text and experiments. JX prepared Table 4. KL prepared Table 5. All authors reviewed the manuscript.

Corresponding author

Correspondence to Kunpeng Zheng.

Ethics declarations

Conflict of interest

Not applicable.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yan, L., Zheng, K., Xia, J. et al. LMix: regularization strategy for convolutional neural networks. SIViP 17, 1245–1253 (2023). https://doi.org/10.1007/s11760-022-02332-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02332-x

Keywords

Search

Navigation