Skip to main content
SpringerLink
Log in

Few-shot defect detection using feature enhancement and image generation for manufacturing quality inspection

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Visual defect detection, which is pivotal in industrial quality control, often requires extensive datasets for training deep-learning models. However, in industrial environments, the presence of multiple production batches, small lot sizes, and rapidly evolving task requirements make it challenging to acquire sufficient and diverse defect data. To address these challenges, this study introduces an innovative approach to data augmentation and defect detection under few-data conditions. Our strategy employs a two-stage feature-enhancement method complemented by an expert knowledge-based image-generation method. The former ensures robustness in color and richness in feature representation by focusing on local defect irregularities. The latter leverages both the core mechanistic understanding of experts and nonquantitative empirical insights, combined with a variable background domain, to generate rich images with diverse defect features. We evaluated the feasibility and effectiveness of the proposed method using a proprietary dataset curated by an automotive component manufacturing enterprise. The experimental results revealed that, across the five defect categories, training on 2 images achieved an F1-Score of 41.06%, which improved to 67.11% when the training dataset was expanded to 15 images. A comparative analysis with prevailing feature enhancement and image generation methods revealed our solution’s superiority, bridging methodological drawbacks, and markedly eclipsing others in terms of enhancement potency. Furthermore, when benchmarked against additional public industrial datasets, the model exhibited exceptional adaptability and superior performance, thus highlighting its potential as a robust tool for quality control in manufacturing scenarios.

Graphical Abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1:
Fig. 5
Algorithm 2:
Fig. 6
Algorithm 3:
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Data availability

The data that support the findings of this study are available on request from the corresponding author XQ W. The data are not publicly available due to them containing information that could compromise the privacy of research participating companies.

References

  1. Escobar CA, McGovern ME, Morales-Menendez R (2021) Quality 4.0: a review of big data challenges in manufacturing. J Intell Manuf 32:2319–2334. https://doi.org/10.1007/s10845-021-01765-4

    Article  Google Scholar 

  2. Gao Y, Li X, Wang XV et al (2022) A review on recent advances in vision-based defect recognition towards Industrial Intelligence. J Manuf Syst 62:753–766. https://doi.org/10.1016/j.jmsy.2021.05.008

    Article  Google Scholar 

  3. Shang H, Sun C, Liu J et al (2023) Defect-aware transformer network for intelligent visual surface defect detection. Adv Eng Inform 55:101882. https://doi.org/10.1016/j.aei.2023.101882

    Article  Google Scholar 

  4. Chow JK, Su Z, Wu J et al (2020) Anomaly detection of defects on concrete structures with the convolutional autoencoder. Adv Eng Inform 45:101105. https://doi.org/10.1016/j.aei.2020.101105

    Article  Google Scholar 

  5. Alzubaidi L, Zhang J, Humaidi AJ et al (2021) Review of deep learning: concepts, CNN architectures, challenges, applications, future directions. J Big Data 8:53. https://doi.org/10.1186/s40537-021-00444-8

    Article  Google Scholar 

  6. Ren R, Hung T, Tan KC (2018) A generic deep-learning-based approach for automated surface inspection. IEEE Trans Cybern 48:929–940. https://doi.org/10.1109/TCYB.2017.2668395

    Article  Google Scholar 

  7. Zhang X, Cai F, Hu X et al (2022) A contrastive learning-based task adaptation model for few-shot intent recognition. Inf Process Manag 59:102863. https://doi.org/10.1016/j.ipm.2021.102863

    Article  Google Scholar 

  8. Singh SA, Kumar AS, Desai KA (2023) Comparative assessment of common pre-trained CNNs for vision-based surface defect detection of machined components. Expert Syst Appl 218:119623. https://doi.org/10.1016/j.eswa.2023.119623

    Article  Google Scholar 

  9. de la López F, Gómez-Sirvent JL, Sánchez-Reolid R et al (2022) Geometric transformation-based data augmentation on defect classification of segmented images of semiconductor materials using a ResNet50 convolutional neural network. Expert Syst Appl 206:117731. https://doi.org/10.1016/j.eswa.2022.117731

    Article  Google Scholar 

  10. Xu Y, Wei S, Bao Y, Li H (2019) Automatic seismic damage identification of reinforced concrete columns from images by a region-based deep convolutional neural network. Struct Control Health Monit 26:e2313. https://doi.org/10.1002/stc.2313

    Article  Google Scholar 

  11. Kaur H, Pannu HS, Malhi AK (2020) A systematic review on imbalanced data challenges in machine learning: applications and solutions. ACM Comput Surv 52:1–36. https://doi.org/10.1145/3343440

    Article  Google Scholar 

  12. Yousefan M, Najafabadi HE, Amirkhani H et al (2022) Deep anomaly detection in hyperspectral images based on membership maps and object area filtering. Expert Syst Appl 191:116200. https://doi.org/10.1016/j.eswa.2021.116200

    Article  Google Scholar 

  13. Bau D, Zhu JY, Wulff J et al (2019) Seeing What a GAN Cannot Generate. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 4502–4511

  14. Yao H, Zhang C, Wei Y et al (2020) Graph few-shot learning via knowledge transfer. AAAI 34:6656–6663. https://doi.org/10.1609/aaai.v34i04.6142

    Article  Google Scholar 

  15. Zhou C, Gu Y, Fang G, Lin Z (2022) Automatic morphological classification of galaxies: convolutional autoencoder and bagging-based multiclustering model. AJ 163:86. https://doi.org/10.3847/1538-3881/ac4245

    Article  Google Scholar 

  16. Hospedales TM, Antoniou A, Micaelli P, Storkey AJ (2021) Meta-learning in neural networks: a Survey. IEEE Trans Pattern Anal Mach Intell 1–1:1. https://doi.org/10.1109/TPAMI.2021.3079209

    Article  Google Scholar 

  17. Wu Y, Lin Y, Dong X et al (2018) Exploit the unknown gradually: one-shot video-based person re-identification by Stepwise Learning. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Salt Lake City, UT, pp 5177–5186

  18. Xu Z, Zhu L, Yang Y (2017) Few-shot object recognition from machine-labeled web images. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Honolulu, HI, pp 5358–5366

  19. Wang T, Zhang X, Yuan L, Feng J (2019) Few-shot adaptive faster R-CNN. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 7166–7175

  20. Wang K, Liew JH, Zou Y et al (2019) PANet: few-shot image semantic segmentation with prototype alignment. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Seoul, Korea (South), pp 9196–9205

  21. Yong G, Jeon K, Gil D, Lee G (2022) Prompt engineering for zero-shot and few-shot defect detection and classification using a visual-language pretrained model. Comput Aided Civil Eng Mice 12954. https://doi.org/10.1111/mice.12954

  22. Wang W, Mi C, Wu Z et al (2022) A real-time steel surface defect detection approach with high accuracy. IEEE Trans Instrum Meas 71:1–10. https://doi.org/10.1109/TIM.2021.3127648

    Article  Google Scholar 

  23. Sheynin S, Benaim S, Wolf L (2021) A hierarchical transformation-discriminating generative model for few shot anomaly detection. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Montreal, QC, Canada, pp 8475–8484

  24. Lu Y, Zheng H, Chand S et al (2022) Outlook on human-centric manufacturing towards industry 5.0. J Manuf Syst 62:612–627. https://doi.org/10.1016/j.jmsy.2022.02.001

    Article  Google Scholar 

  25. Jarrahi MH (2018) Artificial intelligence and the future of work: human-AI symbiosis in organizational decision making. Bus Horiz 61:577–586. https://doi.org/10.1016/j.bushor.2018.03.007

    Article  Google Scholar 

  26. Bay H, Ess A, Tuytelaars T, Van Gool L (2008) Speeded-Up robust features (SURF). Comput Vis Image Underst 110:346–359. https://doi.org/10.1016/j.cviu.2007.09.014

    Article  Google Scholar 

  27. Pasadas DJ, Ramos HG, Feng B et al (2020) Defect classification with SVM and wideband excitation in Multilayer Aluminum plates. IEEE Trans Instrum Meas 69:241–248. https://doi.org/10.1109/TIM.2019.2893009

    Article  Google Scholar 

  28. Cheng X, Yu J (2021) RetinaNet with difference channel attention and adaptively spatial feature fusion for steel surface defect detection. IEEE Trans Instrum Meas 70:1–11. https://doi.org/10.1109/TIM.2020.3040485

    Article  Google Scholar 

  29. Jin X, Wang Y, Zhang H et al (2020) DM-RIS: deep multimodel rail inspection system with improved MRF-GMM and CNN. IEEE Trans Instrum Meas 69:1051–1065. https://doi.org/10.1109/TIM.2019.2909940

    Article  Google Scholar 

  30. Zhang Y, Wang W, Li Z et al (2023) Development of a cross-scale weighted feature fusion network for hot-rolled steel surface defect detection. Eng Appl Artif Intell 117:105628. https://doi.org/10.1016/j.engappai.2022.105628

    Article  Google Scholar 

  31. Hao R, Lu B, Cheng Y et al (2021) A steel surface defect inspection approach towards smart industrial monitoring. J Intell Manuf 32:1833–1843. https://doi.org/10.1007/s10845-020-01670-2

    Article  Google Scholar 

  32. Jain S, Seth G, Paruthi A et al (2022) Synthetic data augmentation for surface defect detection and classification using deep learning. J Intell Manuf 33:1007–1020. https://doi.org/10.1007/s10845-020-01710-x

    Article  Google Scholar 

  33. Yang B, Liu Z, Duan G, Tan J (2022) Mask2Defect: a prior knowledge-based data augmentation method for metal surface defect inspection. IEEE Trans Ind Inf 18:6743–6755. https://doi.org/10.1109/TII.2021.3126098

    Article  Google Scholar 

  34. Guo R, Liu H, Xie G, Zhang Y (2021) Weld defect detection from imbalanced radiographic images based on contrast enhancement conditional generative adversarial network and transfer learning. IEEE Sens J 21:10844–10853. https://doi.org/10.1109/JSEN.2021.3059860

    Article  Google Scholar 

  35. Niu S, Li B, Wang X, Lin H (2020) Defect image sample generation with GAN for improving defect recognition. IEEE Trans Automat Sci Eng 1–12:1. https://doi.org/10.1109/TASE.2020.2967415

    Article  Google Scholar 

  36. Zhou F, Yang S, Fujita H et al (2020) Deep learning fault diagnosis method based on global optimization GAN for unbalanced data. Knowl Based Syst 187:104837. https://doi.org/10.1016/j.knosys.2019.07.008

    Article  Google Scholar 

  37. Tsai D-M, Fan S-KS, Chou Y-H (2021) Auto-annotated deep segmentation for surface defect detection. IEEE Trans Instrum Meas 70:1–10. https://doi.org/10.1109/TIM.2021.3087826

    Article  Google Scholar 

  38. Yan L, Zheng Y, Cao J (2018) Few-shot learning for short text classification. Multimed Tools Appl 77:29799–29810. https://doi.org/10.1007/s11042-018-5772-4

    Article  Google Scholar 

  39. Yu R, Guo B, Yang K (2022) Selective prototype network for few-shot metal surface defect segmentation. IEEE Trans Instrum Meas 71:1–10. https://doi.org/10.1109/TIM.2022.3196447

    Article  Google Scholar 

  40. Xiao W, Song K, Liu J, Yan Y (2022) Graph embedding and optimal transport for few-shot classification of metal surface defect. IEEE Trans Instrum Meas 71:1–10. https://doi.org/10.1109/TIM.2022.3169547

    Article  Google Scholar 

  41. Song Y, Liu Z, Ling S et al (2022) Coarse-to-fine few-shot defect recognition with dynamic weighting and joint metric. IEEE Trans Instrum Meas 71:1–10. https://doi.org/10.1109/TIM.2022.3193204

    Article  Google Scholar 

  42. Yun S, Han D, Chun S et al (2019) CutMix: regularization strategy to train strong classifiers with localizable features. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Seoul, Korea (South), pp 6022–6031

  43. Zhong Z, Zheng L, Kang G et al (2020) Random erasing data Augmentation. AAAI 34:13001–13008. https://doi.org/10.1609/aaai.v34i07.7000

    Article  Google Scholar 

  44. Arjovsky M, Chintala S, Bottou L (2017) Wasserstein generative adversarial networks. In: International Conference on Machine Learning, pp. 214–223. PMLR

  45. Zhang H, Goodfellow I, Metaxas D, Odena A (2019) Self-attention generative adversarial networks. In: International conference on machine learning, pp. 7354–7363. PMLR

Download references

Acknowledgements

This research is supported by National Social Science Fund of China [20BGL108].

Author information

Authors and Affiliations

  1. School of Mechanical Engineering, Hefei University of Technology, Hefei, 230009, China

    Yu Gong, Mingzhou Liu, Xiaoqiao Wang & Jing Hu

  2. School of Mechanical and Electronic Engineering, Suzhou University, Suzhou, 234000, China

    Conghu Liu

Authors
  1. Yu Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingzhou Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoqiao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Conghu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jing Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization: Yu Gong, Xiaoqiao Wang; Data curation: Y u Gong, Conghu Liu; Investigation: Xiaoqiao Wang, Mingzhou Liu; Methodology: Yu Gong, Xiaoqiao Wang; Project administration: Mingzhou Liu, Conghu Liu; Supervision: Xiaoqiao Wang, Jing Hu; V alidation: Jing Hu; Writing – original draft: Yu Gong; Writing – review & editing: Xiaoqiao Wang.

Corresponding author

Correspondence to Xiaoqiao Wang.

Ethics declarations

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gong, Y., Liu, M., Wang, X. et al. Few-shot defect detection using feature enhancement and image generation for manufacturing quality inspection. Appl Intell 54, 375–397 (2024). https://doi.org/10.1007/s10489-023-05199-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05199-8

Keywords

Search

Navigation