Skip to main content
SpringerLink
Log in

Deep learning-based automated steel surface defect segmentation: a comparative experimental study

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The use of machine vision and deep learning for intelligent industrial inspection has become increasingly important in automating the production processes. Despite the fact that machine vision approaches are used for industrial inspection, deep learning-based defect segmentation has not been widely studied. While state-of-the-art segmentation methods are often tuned for a specific purpose, extending them to unknown sets or other datasets, such as defect segmentation datasets, require further analysis. In addition, recent contributions and improvements in image segmentation methods have not been extensively investigated for defect segmentation. To address these problems, we conducted a comparative experimental study on several recent state-of-the-art deep learning-based segmentation methods for steel surface defect segmentation and evaluated them on the basis of segmentation performance, processing time, and computational complexity using two public datasets, NEU-Seg and Severstal Steel Defect Detection (SSDD). In addition we proposed and trained a hybrid transformer-based encoder with CNN-based decoder head and achieved state-of-the-art results, a Dice score of 95.22% (NEU-Seg) and 95.55% (SSDD).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Data Availability

The datasets that are used for this experimental study are publicly available steel surface defect segmentation datasets. The NEU-Seg dataset if from [10] and the SDDD dataset can be accessed from [39]. In addition, the specific partition of the dataset used during training and experimentation can be provided up on request.

Code Availability

The implementation code for this study including the trained weights will soon be available at: https://github.com/djene-mengistu/dseg_models

References

  1. Abdou M A (2022) Literature review: efficient deep neural networks techniques for medical image analysis. Neural Comput Appl 34(8):5791–5812

    Article  Google Scholar 

  2. Abu M, Amir A, Lean YH et al (2021) The performance analysis of transfer learning for steel defect detection by using deep learning. In: J Phys: Conf Series. IOP Publishing, p 012041

  3. Ahmed K R (2023) Dsteelnet: a real-time parallel dilated cnn with atrous spatial pyramid pooling for detecting and classifying defects in surface steel strips. Sensors 23(1):544

    Article  Google Scholar 

  4. Badrinarayanan V, Kendall A, Cipolla R (2016) Segnet: a deep convolutional encoder-decoder architecture for image segmentation. arXiv:1511.00561 [cs]

  5. Bozkurt F (2022) Skin lesion classification on dermatoscopic images using effective data augmentation and pre-trained deep learning approach. Multimed Tools Applic, 1–19

  6. Chen L-C, Papandreou G, Schroff F et al (2017) Rethinking atrous convolution for semantic image segmentation. arXiv:1706.05587

  7. Chen L-C, Zhu Y, Papandreou G et al (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 833–851, DOI https://doi.org/10.1007/978-3-030-01234-2_49

  8. Damacharla P, Rao A, Ringenberg J et al (2021) Tlu-net: a deep learning approach for automatic steel surface defect detection. In: 2021 International Conference on Applied Artificial Intelligence (ICAPAI). IEEE, pp 1–6

  9. Demir K, Ay M, Cavas M et al (2022) Automated steel surface defect detection and classification using a new deep learning-based approach. Neural Comput Appl, 1–18

  10. Dong H, Song K, He Y et al (2020) Pga-net: pyramid feature fusion and global context attention network for automated surface defect detection. IEEE Trans Industr Inf 16(12):7448–7458. https://doi.org/10.1109/TII.2019.2958826

    Article  Google Scholar 

  11. Dosovitskiy A, Beyer L, Kolesnikov A et al (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv:2010.11929

  12. Elizar E, Zulkifley M A, Muharar R et al (2022) A review on multiscale-deep-learning applications. Sensors 22(19):7384

    Article  Google Scholar 

  13. Fu G, Sun P, Zhu W et al (2019) A deep-learning-based approach for fast and robust steel surface defects classification. Opt Lasers Eng 121:397–405. https://doi.org/10.1016/j.optlaseng.2019.05.005

    Article  Google Scholar 

  14. Fu J, Liu J, Tian H et al (2019) Dual attention network for scene segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3146–3154

  15. Gao Y, Gao L, Li X et al (2020) A semi-supervised convolutional neural network-based method for steel surface defect recognition. Robot Comput-Integr Manuf 61:101825. https://doi.org/10.1016/j.rcim.2019.101825

    Article  Google Scholar 

  16. Hao R, Lu B, Cheng Y et al (2020) A steel surface defect inspection approach towards smart industrial monitoring. J Intell Manuf. https://doi.org/10.1007/s10845-020-01670-2, [Online; accessed 2021-05-17]

  17. He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Las Vegas, pp 770–778, DOI https://doi.org/10.1109/CVPR.2016.90

  18. He Y, Song K, Meng Q et al (2020) An end-to-end steel surface defect detection approach via fusing multiple hierarchical features. IEEE Trans Instrum Meas 69(4):1493–1504. https://doi.org/10.1109/TIM.2019.2915404

    Article  Google Scholar 

  19. Huang Z, Wang X, Huang L et al (2019) Ccnet: criss-cross attention for semantic segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 603–612

  20. Huang Y, Qiu C, Wang X et al (2020) A compact convolutional neural network for surface defect inspection. Sensors 20(7):1974. https://doi.org/10.3390/s20071974

    Article  Google Scholar 

  21. Huang Z, Wu J, Xie F (2021) Automatic surface defect segmentation for hot-rolled steel strip using depth-wise separable u-shape network. Mater Lett 301:130271. https://doi.org/10.1016/j.matlet.2021.130271

    Article  Google Scholar 

  22. Kirillov A, Girshick R, He K et al (2019) Panoptic feature pyramid networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Long Beach, pp 6392–6401, DOI https://doi.org/10.1109/CVPR.2019.00656

  23. Liu Z, Yang B, Duan G et al (2020) Visual defect inspection of metal part surface via deformable convolution and concatenate feature pyramid neural networks. IEEE Trans Instrum Meas 69(12):9681–9694. https://doi.org/10.1109/TIM.2020.3001695

    Article  Google Scholar 

  24. Liu Z, Lin Y, Cao Y et al (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Montreal, pp 9992–10002, DOI https://doi.org/10.1109/ICCV48922.2021.00986

  25. Liu Z, Mao H, Wu C-Y et al (2022) A convnet for the 2020s. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11976–11986

  26. Liu Z, Zeng Z, Li J, Teng S (2022) Automatic detection and quantification of hot-rolled steel surface defects using deep learning. Arab J Sci Eng, 1–13

  27. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. CVPR2015, 10

  28. Luo Q, Fang X, Su J et al (2020) Automated visual defect classification for flat steel surface: a survey. IEEE Trans Instrum Meas 69(12):9329–9349. https://doi.org/10.1109/TIM.2020.3030167

    Article  Google Scholar 

  29. Luo Q, Fang X, Sun Y et al (2019) Surface defect classification for hot-rolled steel strips by selectively dominant local binary patterns. IEEE Access 7:23488–23499. https://doi.org/10.1109/ACCESS.2019.2898215

    Article  Google Scholar 

  30. Ma J, Wang Y, Shi C, Lu C (2018) Fast surface defect detection using improved gabor filters. In: 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, pp 1508–1512, DOI https://doi.org/10.1109/ICIP.2018.8451351, (to appear in print)

  31. Mehta S, Rastegari M, Caspi A et al (2018) Espnet: efficient spatial pyramid of dilated convolutions for semantic segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV), DOI https://doi.org/10.1007/978-3-030-01249-6_34

  32. Paszke A, Chaurasia A, Kim S et al (2016) Enet: a deep neural network architecture for real-time semantic segmentation. arXiv:1606.02147 [cs]

  33. Peng C, Zhang X, Yu G et al (2017) Large kernel matters – improve semantic segmentation by global convolutional network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), DOI https://doi.org/10.1109/CVPR.2017.189

  34. Prappacher N, Bullmann M, Bohn G et al (2020) Defect detection on rolling element surface scans using neural image segmentation. Appl Sci 10(9):3290. https://doi.org/10.3390/app10093290

    Article  Google Scholar 

  35. Qian K (2019) Automated detection of steel defects via machine learning based on real-time semantic segmentation. In: Proceedings of the 3rd international conference on video and image processing , pp 42–46

  36. Ren Z, Fang F, Yan N et al (2021) State of the art in defect detection based on machine vision. International Journal of Precision Engineering and Manufacturing-Green Technology. https://doi.org/10.1007/s40684-021-00343-6

  37. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention – MICCAI 2015, vol 9351. Springer International Publishing, pp 234–241, DOI https://doi.org/10.1007/978-3-319-24574-4_28

  38. Sandler M, Howard A, Zhu M et al (June 2018) Mobilenetv2: inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF Conference on computer vision and pattern recognition. IEEE, Salt Lake City, pp 4510–4520, DOI https://doi.org/10.1109/CVPR.2018.00474

  39. Severstal (2019) Severstal: Steel defect detection, kaggle challange 2019. https://www.kaggle.com/c/severstal-steel-defect-detection

  40. Sime D M, Wang G, Zeng Z et al (2022) Semi-supervised defect segmentation with pairwise similarity map consistency and ensemble-based cross-pseudo labels. IEEE Trans Industr Inf. https://doi.org/10.1109/TII.2022.3230785

  41. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  42. Suvdaa B, Ahn J, Ko J (2012) Steel surface defects detection and classification using sift and voting strategy. Int J Softw Eng Applic 6(2):6

    Google Scholar 

  43. Tabernik D, Šela S, Skvarč J et al (2020) Segmentation-based deep-learning approach for surface-defect detection. J Intell Manuf 31 (3):759–776. https://doi.org/10.1007/s10845-019-01476-x

    Article  Google Scholar 

  44. Tan M, Le Q (2019) Efficientnet: rethinking model scaling for convolutional neural networks. In: International conference on machine learning. PMLR, pp 6105–6114

  45. Tang B, Chen L, Sun W et al (2022) Review of surface defect detection of steel products based on machine vision. IET Image Proc

  46. Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. Advances in neural information processing systems, 30

  47. Wang J, Sun K, Cheng T et al (2021) Deep high-resolution representation learning for visual recognition. IEEE Trans Pattern Anal Mach Intell 43 (10):3349–3364. https://doi.org/10.1109/TPAMI.2020.2983686

    Article  Google Scholar 

  48. Wang J, Zhang Q, Liu G (2022) Drcdct-net: a steel surface defect diagnosis method based on a dual-route cross-domain convolution-transformer network. Meas Sci Technol 33(9):095404

    Article  Google Scholar 

  49. Wang W, Xie E, Li X et al (2022) Pvt v2: improved baselines with pyramid vision transformer. Comput Vis Media 8(3):415–424. https://doi.org/10.1007/s41095-022-0274-8

    Article  Google Scholar 

  50. Wanin M (1993) In-line metallurgical process control in the steel industry. Le Journal de Physique IV 03(C7):C7–1101–C7–1107. https://doi.org/10.1051/jp4:19937172

    Article  Google Scholar 

  51. Wu H, Zhang J, Huang K et al (2019) Fastfcn: rethinking dilated convolution in the backbone for semantic segmentation. arXiv:1903.11816 [cs]

  52. Wu T, Tang S, Zhang R et al (2021) Cgnet: a light-weight context guided network for semantic segmentation. IEEE Trans Image Process 30:1169–1179. https://doi.org/10.1109/TIP.2020.3042065

    Article  Google Scholar 

  53. Xiao T, Liu Y, Zhou B et al (2018) Unified perceptual parsing for scene understanding. In: Proceedings of the European conference on computer vision (ECCV), pp 418–434, DOI https://doi.org/10.1007/978-3-030-01228-1_26

  54. Xie S, Girshick R, Dollar P et al (2017) Aggregated residual transformations for deep neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, pp 5987–5995, DOI https://doi.org/10.1109/CVPR.2017.634

  55. Xie E, Wang W, Yu Z et al (2021) Segformer: simple and efficient design for semantic segmentation with transformers. In: 35th Conference on Neural Information Processing Systems (NeurIPS 2021), p 14

  56. Xu R, Hao R, Huang B (2022) Efficient surface defect detection using self-supervised learning strategy and segmentation network. Adv Eng Inform 52:101566

    Article  Google Scholar 

  57. Yan H, Zhang C, Wu M (2022) Lawin transformer: improving semantic segmentation transformer with multi-scale representations via large window attention. arXiv:2201.01615

  58. Yu F, Koltun V (2016) Multi-scale context aggregation by dilated convolutions. arXiv:1511.07122 [cs]

  59. Yu C, Wang J, Peng C et al (2018) Bisenet: bilateral segmentation network for real-time semantic segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV)

  60. Yuan Y, Huang L, Guo J et al (2021) Ocnet: object context for semantic segmentation. Int J Comput Vision 129(8):2375–2398. https://doi.org/10.1007/s11263-021-01465-9

    Article  Google Scholar 

  61. Zhang H, Dana K, Shi J et al (2018) Context encoding for semantic segmentation. In: 2018 IEEE/CVF Conference on computer vision and pattern recognition, Salt Lake City, pp 7151–7160, DOI https://doi.org/10.1109/CVPR.2018.00747

  62. Zhang C, Patras P, Haddadi H (2019) Deep learning in mobile and wireless networking: a survey. IEEE Commun Surv Tutor 21(3):2224–2287

    Article  Google Scholar 

  63. Zhang H, Goodfellow I, Metaxas D et al (2019) Self-attention generative adversarial networks. In: Proceedings of the 36th International Conference on Machine Learning, Long Beach, pp 7354–7363

  64. Zhang Q, Yang Y-B (2021) Rest: an efficient transformer for visual recognition. In: Advances in neural information processing systems, vol 34. Curran Associates, Inc., pp 15475–15485

  65. Zhao H, Shi J, Qi X et al (2017) Pyramid scene parsing network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Honolulu, pp 6230–6239, DOI https://doi.org/10.1109/CVPR.2017.660

  66. Zhao H, Qi X, Shen X et al (2018) Icnet for real-time semantic segmentation on high-resolution images. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 418–434

  67. Zheng S, Lu J, Zhao H et al (2021) Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, pp 6877–6886, DOI https://doi.org/10.1109/CVPR46437.2021.00681

  68. Zheng X, Zheng S, Kong Y et al (2021) Recent advances in surface defect inspection of industrial products using deep learning techniques. Int J Adv Manuf Technol 113(1-2):35–58. https://doi.org/10.1007/s00170-021-06592-8

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by Natural Science Foundation of China (No. 51975107, No.52175292), Science and Technology Project of Sichuan Province (No. 2020ZDZX0015).

Author information

Authors and Affiliations

  1. School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, Sichuan, China

    Dejene M. Sime, Guotai Wang, Zhi Zeng & Bei Peng

Authors
  1. Dejene M. Sime
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guotai Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhi Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bei Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bei Peng.

Ethics declarations

Ethics approval

Not applicable

Conflict of Interests

The authors declare that they have no known competing interests and all the materials and methods used in the study are duly acknowledged.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sime, D.M., Wang, G., Zeng, Z. et al. Deep learning-based automated steel surface defect segmentation: a comparative experimental study. Multimed Tools Appl 83, 2995–3018 (2024). https://doi.org/10.1007/s11042-023-15307-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15307-y

Keywords

Search

Navigation