Skip to main content
SpringerLink
Log in

Multi-level joint distributed alignment-based domain adaptation for cross-scenario strip defect recognition

  • Published:
Journal of Intelligent Manufacturing Aims and scope Submit manuscript

Abstract

Deep learning-based methods for object recognition under closed situation have achieved remarkable progress. However, the data distribution of cross-scenario images of strip steel defects varies considerably due to imaging factors such as camera type, parameters, and unstable illumination in different scenarios. Large domain gap and limited diversity may jeopardize the generalization of model over the cross-scenario target domain. In this article, we propose a novel Multi-Level Joint Distribution Alignment Domain Adaptation model (MLJDA). The model unites different granularity perspectives from pixel-, feature-, and category-level to align cross-domain data. First, we propose a parameter sharing style-expansion module based on pixel-level feature mapping, which synthesizes multi-source domain data with different distributions compared to source domain to improve the generalization performance of model towards unknown scenarios and synthesizes similar multi-target domain data. Second, the generated multi-source domain and multi-target domain data are subjected to feature-level adversarial learning to obtain domain-invariant representations. Finally, we design multi-domain collaborative pseudo-label verification based on a voting mechanism to purify cross-scenario target domain category labels. Experimental results on the Handan Steel Defect Dataset and the public Severstal Steel Defect Dataset show that MLJDA can effectively solve the problem of recognizing strip steel defects across scenarios. The MLJDA achieve a F1 score of 98\(\%\) and 90\(\%\) on the Handan Steel Defect Dataset and Severstal Steel Defect, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data Availability

Severstal Steel Defect Dataset is openly available in a public repository that does not issue DOIs. The Severstal Steel Defect Dataset that supports the findings of this study is openly available at [Severstal teams] at [https://www.kaggle.com/c/severstal-steel-defect-detection/data].

References

  • Boschetto, A., Bottini, L., & Vatanparast, S. (2023). Powder bed monitoring via digital image analysis in additive manufacturing. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-023-02091-7

    Article  Google Scholar 

  • Boudani, F. Z., Nacereddine, N., & Laiche, N. (2021). Content-Based Image Retrieval for Surface Defects of Hot Rolled Steel Strip Using Wavelet-Based LBP. In Progress in Artificial Intelligence and Pattern Recognition: 7th International Workshop on Artificial Intelligence and Pattern Recognition, IWAIPR 2021, Havana, Cuba. pp. 404–413. https://doi.org/10.1007/978-3-030-89691-1_39

  • Chu, T., Liu, Y., Deng, J., Li, W., & Duan, L. (2022, June). Denoised Maximum Classi?er Discrepancy for Source-Free Unsupervised Domain Adaptation. Proceedings of the AAAI Conference on Artificial Intelligence 36(1), 472–480. https://doi.org/10.1609/aaai.v36i1.19925

  • Cui, S., Wang, S., Zhuo, J., Su, C., Huang, Q., & Tian, Q. (2020). Gradually vanishing bridge for adversarial domain adaptation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA (pp. 12452–12461). https://doi.org/10.1109/CVPR42600.2020.01247

  • Czimmermann, T., Ciuti, G., Milazzo, M., Chiurazzi, M., Roccella, S., Oddo, C. M., & Dario, P. (2020). Visual-based defect detection and classification approaches for industrial applications: A survey. Sensors, 20(5), 1459. https://doi.org/10.3390/s20051459

    Article  Google Scholar 

  • Fan, R., Wang, H., Bocus, M. J., & Liu, M. (2020). We learn better road pothole detection: from attention aggregation to adversarial domain adaptation. In Computer VisionCECCV 2020 Workshops(ECCV 2022), Glasgow, UK (pp. 285–300). https://doi.org/10.1007/978-3-030-66823-5_17

  • Ganin, Y., & Lempitsky, V. (2015). Unsupervised domain adaptation by backpropagation. In Proceedings of the 32nd International Conference on Machine Learning, PMLR, Lille, France (pp. 1180–1189). https://doi.org/10.48550/arXiv.1409.7495

  • Ganin, Y., & Lempitsky, V. (2015). Unsupervised domain adaptation by backpropagation. In Proceedings of the 32nd International Conference on Machine Learning, PMLR, Lille, France (pp. 1180–1189). https://doi.org/10.48550/arXiv.1409.7495

  • Gan, J., Li, Q., Wang, J., & Yu, H. (2017). A hierarchical extractor-based visual rail surface inspection system. IEEE Sensors Journal, 17(23), 7935–7944. https://doi.org/10.1109/JSEN.2017.2761858

    Article  Google Scholar 

  • Goetz, A., Durmaz, A. R., Mller, M., et al. (2022). Addressing materials microstructure diversity using transfer learning. NPJ Computer Materials, 8, 27. https://doi.org/10.1038/s41524-022-00703-z

    Article  Google Scholar 

  • Han, C., Li, G., & Liu, Z. (2022). Two-stage edge reuse network for salient object detection of strip steel surface defects. IEEE Transactions on Instrumentation and Measurement, 71, 1–12. https://doi.org/10.1109/TIM.2022.3200114

    Article  Google Scholar 

  • Hao, R., Lu, B., Cheng, Y., Li, X., & Huang, B. (2021). A steel surface defect inspection approach towards smart industrial monitoring. Journal of Intelligent Manufacturing, 32, 1833–1843. https://doi.org/10.1007/s10845-020-01670-2

    Article  Google Scholar 

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA (pp. 770–778). https://doi.org/10.1109/CVPR.2016.90

  • Huang, X., & Belongie, S. (2017). Arbitrary style transfer in real-time with adaptive instance normalization. In 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy (pp. 1510–1519). https://doi.org/10.1109/ICCV.2017.167

  • Hu, L., Duan, F., Ding, K., & Ye, S. H. (2005). Research on surface defects on line detection system for steel plate using computer vision. Kang T’ieh/Iron and Steel (Peking), 40(2), 59–61.

    Google Scholar 

  • Hu, H., & Zhu, Z. (2023). Sim-YOLOv5s: A method for detecting defects on the end face of lithium battery steel shells. Advanced Engineering Informatics, 55, 101824. https://doi.org/10.1016/j.aei.2022.101824

    Article  Google Scholar 

  • Liang, Y., Xu, K., Zhou, P., & Zhou, D. (2022). Automatic defect detection of texture surface with an efficient texture removal network. Advanced Engineering Informatics, 53, 101672. https://doi.org/10.1016/j.aei.2022.101672

    Article  Google Scholar 

  • Liu, D., Zhang, D., Song, Y., & Zhang, F., et al. (2020). Unsupervised instance segmentation in microscopy images via panoptic domain adaptation and task re-weighting. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA (pp. 4242–4251). https://doi.org/10.1109/CVPR42600.2020.00430

  • Liu, Y., Xiao, H., Xu, J., & Zhao, J. (2022). A rail surface defect detection method based on pyramid feature and lightweight convolutional neural network. IEEE Transactions on Instrumentation and Measurement, 71, 1–10. https://doi.org/10.1109/TIM.2022.3165287

    Article  Google Scholar 

  • Long, M., Cao, Y., Wang, J., & Jordan, M. (2015). Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on Machine Learning, PMLR, Lille, France (pp. 97–105). https://doi.org/10.48550/arXiv.1409.7495

  • Long, M., Zhu, H., Wang, J., & Jordan, M. I. (2017). Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference on Machine Learning, PMLR, Precup, Doina and Teh, Yee Whye (pp. 2208–2217). https://doi.org/10.48550/arXiv.1605.06636

  • Nieniewski, M. (2020). Morphological detection and extraction of rail surface defects. IEEE Transactions on Instrumentation and Measurement, 69(9), 6870–6879. https://doi.org/10.1109/TIM.2020.2975454

    Article  Google Scholar 

  • Oster, S., Breese, P. P., Ulbricht, A., Mohr, G., & Altenburg, S. J. (2023). A deep learning framework for defect prediction based on thermographic in-situ monitoring in laser powder bed fusion. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-023-02117-0

    Article  Google Scholar 

  • Park, J. E., & Kim, Y. K. (2023). Semi-supervised learning for steel surface inspection using magnetic flux leakage signal. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-023-02286-y

    Article  Google Scholar 

  • Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Chintala, S. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.

  • Raab, C., Vath, P., Meier, P., & Schleif, FM. (2021). Bridging Adversarial and Statistical Domain Transfer via Spectral Adaptation Networks. Computer Vision—ACCV 2020, Cham. pp. 457–473. https://doi.org/10.1007/978-3-030-69535-4_28

  • Samsudin, S. S., Arof, H., Harun, S. W., Wahab, A. W. A., & Idris, M. Y. I. (2020). Steel surface defect classification using multi-resolution empirical mode decomposition and LBP. Measurement Science and Technology, 32(1), 015601. https://doi.org/10.1088/1361-6501/abab21

    Article  Google Scholar 

  • Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. In 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy (pp. 618–626). https://doi.org/10.1109/ICCV.2017.74

  • Sindagi, V. A., & Srivastava, S. (2017). Domain adaptation for automatic OLED panel defect detection using adaptive support vector data description. International Journal of Computer Vision, 122(2), 193–211. https://doi.org/10.1007/s11263-016-0953-y

    Article  Google Scholar 

  • Szarski, M., & Chauhan, S. (2022). An unsupervised defect detection model for a dry carbon fiber textile. Journal of Intelligent Manufacturing, 33(7), 2075–2092. https://doi.org/10.1007/s10845-022-01964-7

    Article  Google Scholar 

  • Tsai, D. M., Chen, M. C., Li, W. C., & Chiu, W. Y. (2012). A fast regularity measure for surface defect detection. Machine Vision and Applications, 23, 869–886. https://doi.org/10.1007/s00138-011-0403-3

    Article  Google Scholar 

  • Wang, S., Chen, H., Liu, K., Zhou, Y., & Feng, H. (2023). Meta-FSDet: A meta-learning based detector for few-shot defects of photovoltaic modules. Journal of Intelligent Manufacturing, 34(8), 3413–3427. https://doi.org/10.1007/s10845-022-02001-3

    Article  Google Scholar 

  • Wang, S., Chen, H., Zhang, Z., & Su, B. (2024). Multi-scale feature decoupling and similarity distillation for class-incremental defect detection of photovoltaic cells. Measurement, 225, 113997. https://doi.org/10.1016/j.measurement.2023.113997

    Article  Google Scholar 

  • Wang, C., Chen, H., & Zhao, S. (2023). RERN: Rich edge features refinement detection network for polycrystalline solar cell defect segmentation. IEEE Transactions on Industrial Informatics. https://doi.org/10.1109/TII.2023.3275705

    Article  Google Scholar 

  • Wang, C., Chen, H., Zhao, S., & Rahman, M. R. U. (2022). Efficient and refined deep convolutional features network for the crack segmentation of solar cell electroluminescence images. IEEE Transactions on Semiconductor Manufacturing, 35(4), 610–619. https://doi.org/10.1109/TSM.2022.3197933

    Article  Google Scholar 

  • Wang, Y., Li, X., Gao, Y., Wang, L., & Gao, L. (2021). A new Feature-Fusion method based on training dataset prototype for surface defect recognition. Advanced Engineering Informatics, 50, 101392. https://doi.org/10.1016/j.aei.2021.101392

    Article  Google Scholar 

  • Xiao, N., & Zhang, L. (2021). Dynamic weighted learning for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15242–15251). https://doi.org/10.48550/arXiv.2103.13814

  • Xiao, T., Fan, C., Liu, P., & Liu, H. (2021). Simultaneously improve transferability and discriminability for adversarial domain adaptation. Entropy, 24(1), 44. https://doi.org/10.3390/e24010044

    Article  Google Scholar 

  • Ye, S., Wang, Z., Xiong, P., Xu, X., Du, L., Tan, J., & Wang, W. (2023). Multi-stage few-shot micro-defect detection of patterned OLED panel using defect inpainting and multi-scale Siamese neural network. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-023-02168-3

    Article  Google Scholar 

  • Zaghdoudi, R., Seridi, H., & Ziani, S. (2020). Binary Gabor pattern (BGP) descriptor and principal component analysis (PCA) for steel surface defects classification. In 2020 International Conference on Advanced Aspects of Software Engineering (ICAASE), Constantine, Algeria (pp. 1–7). https://doi.org/10.1109/ICAASE51408.2020.9380108

  • Zhang, C., Cui, J., Wu, J., & Zhang, X. (2023). Attention mechanism and texture contextual information for steel plate defects detection. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-023-02149-6

    Article  Google Scholar 

  • Zhao, S., Chen, H., Wang, C., & Shi, S. (2023). SNCF-Net: Scale-aware neighborhood correlation feature network for hotspot defect detection of photovoltaic farms. Measurement, 206, 112342. https://doi.org/10.1016/j.measurement.2022.112342

    Article  Google Scholar 

  • Zhao, S., Chen, H., Wang, C., & Zhang, Z. (2023). SSN: Shift suppression network for endogenous shift of photovoltaic defect detection. IEEE Transactions on Industrial Informatics. https://doi.org/10.1109/TII.2023.3327572

    Article  Google Scholar 

  • Zhao, S., Chen, H., Wang, C., Zhou, Y., & Zhang, Z. (2024). RGR-Net: Refined graph reasoning network for multi-height hotspot defect detection in photovoltaic farms. Expert Systems with Applications, 245, 123034. https://doi.org/10.1016/j.eswa.2023.123034

    Article  Google Scholar 

  • Zheng, X., Zheng, S., Kong, Y., & Chen, J. (2021). Recent advances in surface defect inspection of industrial products using deep learning techniques. The International Journal of Advanced Manufacturing Technology, 113, 35–58. https://doi.org/10.1007/s00170-021-06592-8

    Article  Google Scholar 

  • Zhou, W., Du, D., Zhang, L., Luo, T., & Wu, Y. (2022). Multi-granularity alignment domain adaptation for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 9581–9590. https://doi.org/10.48550/arXiv.2203.16897

  • Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy (pp. 2242–2251). https://doi.org/10.1109/ICCV.2017.244

Download references

Acknowledgements

This work was supported in part by National Natural Science Foundation of China under Grant 62173124, in part by Natural Science Foundation of Hebei Province under Grant F2022202064, in part by National Natural Science Foundation of China under Grant 62073118.

Author information

Author notes

  1. Kun Liu, Ying Yang, Xiaosong Yang, Jingkai Wang, and Weipeng Liu have contributed equally to this work.

Authors and Affiliations

  1. School of Artifificial Intelligence and Data Science, Hebei University of Technology, Tianjin, 300130, China

    Kun Liu, Ying Yang, Xiaosong Yang, Jingkai Wang, Weipeng Liu & Haiyong Chen

Authors
  1. Kun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ying Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaosong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jingkai Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Weipeng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Haiyong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haiyong Chen.

Ethics declarations

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, K., Yang, Y., Yang, X. et al. Multi-level joint distributed alignment-based domain adaptation for cross-scenario strip defect recognition. J Intell Manuf (2024). https://doi.org/10.1007/s10845-024-02344-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10845-024-02344-z

Keywords

Search

Navigation