Skip to main content
SpringerLink
Log in

A novel micro-defect classification system based on attention enhancement

  • Published:
Journal of Intelligent Manufacturing Aims and scope Submit manuscript

Abstract

A surface micro-defect is characterized by a small size and a susceptibility to noise. Micro-defect detection and classification is very challenging. This paper proposes a Micro-defect classification system based on attention enhancement (MDCS) for solving the detection and classification of micro-defects. We combine defect detection with defect classification in MDSC. Micro-defects classification can be better realized based on the auxiliary task of defect detection. In this system, the aim of attention formation in bionic vision is to guide the system to focus on the target by zooming in and out on micro-defects. To avoid noise interference, an attention module based on trilinear feature confluence has been incorporated. Last but not least, the enhancement process based on the attention map improves the classification ability of micro-defects. As part of comparative experiment, we analyzed data including 19,200 fabric images and 4,800 bamboo images. In the micro-defect classification experiment based on MDCS(ResNet-50), the accuracy of fabric data and bamboo data is 88.2% and 89.4% respectively. Compared with ResNet-50, the classification accuracy (64.8%, 67.7%) is improved by 23.4% and 21.7% respectively. In the object detection experiment of micro-defects based on MDCS (ResNet-50), the accuracy of fabric data and bamboo data is 65.1% mAPs and 63.3% mAPs respectively. Compared with HRDNet, the detection accuracy (59.6% mAPs, 52.2% mAPs) is improved by 5.5% mAPs and 11.1% mAPs respectively. Experimental results demonstrate that the proposed system can counteract the interference caused by noise in small object detection, localize micro-defects accurately, and improve micro-defect classification accuracy significantly.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27

Similar content being viewed by others

Data availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  • Anvar, A., & Cho, Y. I. (2020). Automatic metallic surface defect detection using ShuffleDefectNet. Journal of the Korea Society of Computer and Information, 25, 19–26.

    Google Scholar 

  • Bochkovskiy, A., Wang, C. Y., & Liao, H. (2020). YOLOv4: Optimal speed and accuracy of object detection. CVPR, 1–17. https://doi.org/10.48550/arXiv.2004.10934. Accessed 23 Apr 2020

  • Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018b). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4), 834–848.

    Article  Google Scholar 

  • Chen, S., Tan, X., Wang, B., & Hu, X. (2018a). Reverse attention for salient object detection. In Proceedings of the European conference on computer vision (pp. 234–250).

  • Cheng, K. C., Chen, L. L., Li, J. W., Li, K. S., Tsai, N. C., Wang, S. J., Huang, A. Y., Chou, L., Lee, C. S., Chen, J. E., & Liang, H. C. (2021b). Machine learning based detection method for wafer test induced defects. IEEE Transactions on Semiconductor Manufacturing, 34(2), 161–167.

    Article  Google Scholar 

  • Cheng, P., Wang, H., Stojanovic, V., He, S., Shi, K., Luan, X., Liu, F., & Sun, C. (2021a). Asynchronous fault detection observer for 2-D Markov jump systems. IEEE Transactions on Cybernetics. https://doi.org/10.1109/TCYB.2021.3112699

    Article  Google Scholar 

  • Cheng, P., Wang, H., Stojanovic, V., et al. (2022). Dissipativity-based finite-time asynchronous output feedback control for wind turbine system via a hidden Markov model. International Journal of Systems Science, 3177–3189. https://doi.org/10.1080/00207721.2022.2076171. Accessed 24 May 2022

  • Cohen, N., & Hoshen, Y. (2020). Sub-image anomaly detection with deep pyramid correspondences. CVPR, 1–17. https://doi.org/10.48550/arXiv.2005.02357. Accessed 5 May 2020

  • Defard, T., Setkov, A., Loesch, A., & Audigier, R. (2020). PaDiM: A patch distribution modeling framework for anomaly detection and localization. arXiv. http://arxiv.org/abs/2011.08785. Accessed 17 Nov 2020

  • Dike, H. U., Zhou, Y., Deveerasetty, K. K., & Wu, Q. (2018). Unsupervised learning based on artificial neural network: A review. In Proceedings of the 2018 IEEE international conference on cyborg and bionic systems (CBS), Shenzhen (pp. 322–327).

  • Fu, J., Zheng, H., & Tao, M. (2017). Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition. In IEEE conference on computer vision & pattern recognition. IEEE.

  • He, D., Xu, K., Zhou, P., & Zhou, D. (2019). Surface defect classification of steels with a new semi-supervised learning method. Optics and Lasers in Engineering, 117, 40–48.

    Article  Google Scholar 

  • Hsu, C.Y., Liu, W.C., & Kusiak, A. (2021). Multiple time-series convolutional neural network for fault detection and diagnosis and empirical study in semiconductor manufacturing. Journal of Intelligent Manufacturing, 32, 823.

    Article  Google Scholar 

  • Hu, B., & Wang, J. (2020). Detection of PCB surface defects with improved faster-RCNN and feature pyramid network. IEEE Access, 8, 108335–108345.

    Article  Google Scholar 

  • Hu, T., Xu, J., Huang, C., et al. (2018). Weakly supervised bilinear attention network for fine-grained visual classification. CVPR, 1–8. https://doi.org/10.48550/arXiv.1808.02152. Accessed 6 Aug 2018

  • Jie, H., Li, S., Gang, S. (2018). Squeeze-and-excitation networks. IEEE.

  • Kim, Y., Cho, D., & Lee, J. H. (2021). Wafer defect pattern classification with detecting out-of-distribution. Microelectronics Reliability, 122(4), 114157.

    Article  Google Scholar 

  • Lantzanakis, G., Mitraka, Z., & Chrysoulakis, N. (2020). X-SVM: An extension of C-SVM Algorithm for classification of high-resolution satellite imagery. IEEE Transactions on Geoscience and Remote Sensing, 59, 3805.

    Article  Google Scholar 

  • Lee, K., Maji, S., Ravichandran, A., & Soatto, S. (2019). Meta-learning with differentiable convex optimization. CVPR, 6, 7.

    Google Scholar 

  • Li, A., Zhang, J., Lv, Y., Liu, B., Zhang, T., & Dai, Y. (2021). Uncertainty aware joint salient object and camouflaged object detection. In Proceeding of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10071–10081).

  • Li, L., Ma, W., Li, L., & Lu, C. (2019). Research on detection algorithm for bridge cracks based on deep learning. Acta Automatica Sinica, 45, 1727–1742.

    Google Scholar 

  • Li, W., Wang, L., Huo, J., Shi, Y., Gao, Y., & Luo, J. (2020). Asymmetric distribution measure for few-shot learning. IJCAI. 1, 2, 5, 6, 7, 8.

  • Li, Z., Yang, Y., Liu, X., Zhou, F., Wen, S., & Xu, W. (2017). Dynamic computational time for visual attention. In ICCV (pp. 1199–1209).

  • Lin, S., He, Z., & Sun, L. (2019). Defect enhancement generative adversarial net-work for enlarging data set of microcrack defect. IEEE Access, 7, 148413.

    Article  Google Scholar 

  • Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2016). Feature pyramid networks for object detection. CVPR.

  • Liu, X., Sun, L., & Feng, S. (2021). Incomplete multi-view partial multi-label learning. Applied Intelligence, 3, 3289.

    Google Scholar 

  • Liu, Z., Gao, G., Sun, L., et al. (2020). HRDNet: High-resolution detection network for small objects. CVPR, 1–8. https://doi.org/10.48550/arXiv.2006.07607. Accessed 13 Jun 2020

  • Mao, Y., Zhang, J., Wan, Z., et al. (2021). Transformer transforms salient object detection and camouflaged object detection. CVPR, 1–15. https://doi.org/10.48550/arXiv.2104.10127. Accessed 11 April 2021

  • Mayr, M., Hoffmann, M., Maier, A., & Christlein, V. (2019). Weakly supervised segmentation of cracks on solar cells using normalized Lp norm. In Proceedings of the 2019 IEEE international conference on image processing (ICIP), Taipei (pp. 1885–1889).

  • Özgenel, Ç. F., & Sorguç, A. G. (2018). Performance comparison of pretrained convolutional neural networks on crack detection in buildings. In Proceedings of the international symposium on automation and robotics in construction (IAARC), Berlin (pp. 693–700).

  • Qian, X., Liu, F., Jiao, L., Zhang, X., Guo, Y., Liu, X., & Cui, Y. (2021). Ridgelet-Nets with speckle reduction regularization for SAR image scene classification. IEEE Transactions on Geoscience and Remote Sensing, 59(11), 9290–9306.

    Article  Google Scholar 

  • Roth, K., Pemula, L., Zepeda, J., Schölkopf, B., Brox, T., & Gehler, P. (2021). Towards total recall in industrial anomaly detection. arXiv. http://arxiv.org/abs/2106.08265 . Accessed 15 Jun 2021

  • Schlosser, T., Friedrich, M., Beuth, F., & Kowerko, D. (2022). Improving automated visual fault inspection for semiconductor manufacturing using a hybrid multistage system of deep neural networks. Journal of Intelligent Manufacturing, 33(4), 1099–1123.

    Article  Google Scholar 

  • Shen, J., Chen, P., Su, L., Shi, T., Tang, Z., & Liao, G. (2016). X-ray inspection of TSV defects with self-organizing map network and Otsu algorithm. Microelectronics Reliability, 67, 129–134.

    Article  Google Scholar 

  • Silvén, O., Niskanen, M., & Kauppine, H. (2003). Wood inspection with non-supervised clustering. Machine Vision and Applications, 13, 275–285.

    Article  Google Scholar 

  • Silvestre-Blanes, J., Albero-Albero, T., Miralles, I., Pérez-Llorens, R., & Moreno, J. (2019). A public fabric database for defect detection methods and results. Autex Research Journal , 19, 363–374.

    Article  Google Scholar 

  • Sridharan, N. V., & Sugumaran, V. (2022). Visual fault detection in photovoltaic modules using decision tree algorithms with deep learning features. Energy Sources Part A-Recovery Utilization and Environmental Effects. 1–17. https://doi.org/10.1080/15567036.2021.2020379. Accessed 21 Dec 2021

  • Sun, M., Yuan, Y., Zhou, F., & Ding, E. (2018). Multi attention multi-class constraint for fine-grained image recognition. In ECCV (pp. 805–821).

  • Tao, X., Zhang, D., Wang, Z., Liu, X., Zhang, H., & Xu, D. (2020). Detection of power line insulator defects using aerial images analyzed with convolutional neural networks. In IEEE transactions on systems, man, and cybernetics systems (pp. 1486–1498). IEEE.

    Google Scholar 

  • Wang, G., Han, S., Ding, E., & Huang, D. (2021). Student–teacher feature pyramid matching for unsupervised anomaly detection. arXiv. http://arxiv.org/abs/2103.04257 . Accessed 7 Mar 2021

  • Wang, X., Li, J., Yao, M., He, W., & Qian, Y. (2014). Solar cells surface defects detection based on deep learning. Pattern Recognition and Artificial Intelligence, 27, 517–523.

    Google Scholar 

  • Woo, S., Park, J., Lee, JY., et al. (2018). CBAM: Convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV) (pp. 3–19).

  • Xie, S., Girshick, R., P, Dollár., et al. (2016). Aggregated residual transformations for deep neural networks. IEEE.

  • Xu, K., et al. (2015). Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning (pp. 2048–2057).

  • Xu, Z., Li, X., & Stojanovic, V. (2021). Exponential stability of nonlinear state-dependent delayed impulsive systems with applications. Nonlinear Analysis Hybrid Systems, 42(1), 101088.

    Article  Google Scholar 

  • Yang, F., Choi, W., & Lin, Y. (2016). Exploit all the layers: Fast and accurate CNN object detector with scale dependent pooling and cascaded rejection classifiers. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2129–2137). 1, 2.

  • Ye, W., Li, X., Dong, Y., et al. (2018). Single image surface appearance modeling with self-augmented CNNs and inexact supervision[C]//John Wiley and Sons (pp. 201–211). Wiley.

    Google Scholar 

  • Zhang, Z., Wang, X., & Jung, C. (2019). DCSR: Dilated convolutions for single image super-resolution. IEEE Transactions on Image Processing. 28(4), 1625–1635. https://doi.org/10.1109/TIP.2018.2877483

    Article  Google Scholar 

  • Zheng, H., Fu, J., Mei, T., & Luo, J. (2017). Learning multi-attention convolutional neural network for fine grained image recognition. In ICCV (pp. 5209–5217).

  • Zheng, H., Fu, J., Zha, ZJ., et al. (2019). Looking for the devil in the details: Learning trilinear attention sampling network for fine-grained image recognition. IEEE.

  • Zheng, H., Yang, Y., Sun, X., Wen, C. (2018). Nondestructive detection of anchorage quality of rock bolt based on DS-DBN-SVM. In Proceedings of the 2018 international conference on machine learning and cybernetics (ICMLC), Chengdu (pp. 288–293).

Download references

Funding

This work was supported by the National Key R&D Program of China under Grant Number 2018YFB1309200.

Author information

Authors and Affiliations

  1. Harbin Institute of Technology, Harbin, China

    Song Lin, Zhiyong He & Lining Sun

  2. Soochow University, Soochow, China

    Zhiyong He

Authors
  1. Song Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhiyong He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lining Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhiyong He.

Ethics declarations

Conflict of interest

We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and/or company that could be construed as influencing the position presented in, or the review of, the manuscript entitled “A Novel Micro-Defect Classification System Based on Attention Enhancement”.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, S., He, Z. & Sun, L. A novel micro-defect classification system based on attention enhancement. J Intell Manuf 35, 703–726 (2024). https://doi.org/10.1007/s10845-022-02064-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10845-022-02064-2

Keywords

Search

Navigation