Skip to main content
SpringerLink
Log in

Attention-based deep learning for chip-surface-defect detection

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

Unlike objects (such as cats and dogs) in the ImageNet, the surface defects on chips have a relatively tiny defect areas, yet they contain a large amount of information. The traditional deep learning methods have unsatisfied performance for tiny defects. Therefore, we proposed an object detection network combining attention with YOLOV4 for tiny defect detection, denoted as YOLOV4-SA. The network consists of a feature extraction backbone, a spatial attention module (SAM) and a feature fusion module. The SAM can correct the value of the feature map and highlight the defect areas, which identifies the tiny defects more effectively. To support current and future research, we also constructed a chip-surface-defect dataset, including a real set and a synthetic set. The real set contains non-defective and defective images. The synthetic set was generated by three new defect-generation methods. To the best of our knowledge, this is the first defect dataset in the field of advanced packaging chips. YOLOV4-SA was trained and tested on this dataset. Compared with classical defect detection methods, YOLOV4-SA achieved 78.47 mAP (mean of average precision), which was at least 7% better than other methods. For tiny ink and crack defects, the mAP was improved by 11.92 compared with YOLOV4. Lastly, we also proposed a cascading deployment method which may be useful for industrial applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Availability of data and material

The chip-surface-defect dataset has been uploaded to the BaiDu cloud platform, and can be downloaded via the link https://pan.baidu.com/s/1DsZyyO4ITtsLWqFyGS2KEA.

Code availability

The code cannot be shared at this time due to company secret.

Notes

  1. https://www.kla-tencor.com/products/instruments/defect-inspectors

  2. http://www.visionpro.com/lead-frame-inspection

  3. https://www.nordson.com/en/divisions/select/products

  4. https://pytorch.org/

  5. https://grand-tec.com/Products_Solutions/Semiconductor/

References

  1. Zheng X, Zheng S, Kong Y, Chen J (2021) Recent advances in surface defect inspection of industrial products using deep learning techniques. Int J Adv Manuf Technol 1–24

    Google Scholar 

  2. Tao X, Hou W, Xu D (2020) A survey of surface defect detection methods based on deep learning. Acta Automatica Sinica 1:19

    Google Scholar 

  3. Wang T, Chen Y, Qiao M, Snoussi H (2018) A fast and robust convolutional neural network-based defect detection model in product quality control. Int J Adv Manuf Technol 94:3465–3471

    Article  Google Scholar 

  4. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. Proc IEEE Conf Comput Vis Pattern Recognit 770–778

    Google Scholar 

  5. He K, Gkioxari G, Dollár P, Girshick R (2017) Mask r-cnn. Proc IEEE Int Conf Comput Vis 2961–2969

    Google Scholar 

  6. Bochkovskiy A, Chien-Yao W, Liao HYM (2020) YOLOv4: optimal speed and accuracy of object detection arXiv. arXiv (USA) 17

  7. Wang CY, Bochkovskiy A, Liao HYM (2021) Scaled-yolov4: Scaling cross stage partial network. Proc IEEE/CVF Conf Computer Vis Pattern Recognit 13029–13038

    Google Scholar 

  8. Woo S, Park J, Lee JY, Kweon IS (2018) Cbam: Convolutional block attention module. Proc Eur Conf Comput Vis (ECCV) 3–19

    Google Scholar 

  9. Kyeong K, Kim H (2018) Classification of mixed-type defect patterns in wafer bin maps using convolutional neural networks. IEEE Trans Semicond Manuf 31:395–402

    Article  Google Scholar 

  10. Xiong XH, Liu FJ, Liu Y, Ma LH, Wang Q, Ieee (2014) Research and outlook of wireless flip chip package technology development. Ieee, New York

    Book  Google Scholar 

  11. Liu Z, Qu B (2021) Machine vision based online detection of PCB defect. Microprocess Microsyst 82:103807

    Article  Google Scholar 

  12. Li C, Lan H-Q, Sun Y-N, Wang J-Q (2021) Detection algorithm of defects on polyethylene gas pipe using image recognition. Int J Press Vessels Pip 191:104381

    Article  Google Scholar 

  13. Zhong F, He S, Li B (2017) Blob analyzation-based template matching algorithm for LED chip localization. Int J Adv Manuf Technol 93:55–63

    Article  Google Scholar 

  14. Wang Q, Yang R, Wu C, Liu Y (2021) An effective defect detection method based on improved Generative Adversarial Networks (iGAN) for machined surfaces. J Manuf Process 65:373–381

    Article  Google Scholar 

  15. Shu YF, Li B, Lin H (2021) Quality safety monitoring of LED chips using deep learning-based vision inspection methods. Measurement 168:10

    Article  Google Scholar 

  16. Dimitriou N, Leontaris L, Vafeiadis T, Ioannidis D, Wotherspoon T, Tinker G, Tzovaras D (2020) A Deep Learning framework for simulation and defect prediction applied in microelectronics. Simul Model Pract Theory 100:9

    Article  Google Scholar 

  17. Wu H, Gao WB, Xu XR (2020) Solder Joint Recognition Using Mask R-CNN Method. IEEE Trans Compon Pack Manuf Technol 10:525–530

    Article  Google Scholar 

  18. Stern ML, Schellenberger M (2021) Fully convolutional networks for chip-wise defect detection employing photoluminescence images Efficient quality control in LED manufacturing. J Intell Manuf 32:113–126

    Article  Google Scholar 

  19. Dung CV (2019) Autonomous concrete crack detection using deep fully convolutional neural network. Autom Constr 99:52–58

    Article  Google Scholar 

  20. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. Proc IEEE Conf Comput Vis Pattern Recognit 1:580–587

    Google Scholar 

  21. Cha YJ, Choi W, Suh G, Mahmoudkhani S, Buyukozturk O (2018) Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Comput Aided Civil Infrastruct Eng 33:731–747

    Article  Google Scholar 

  22. Tang S, He F, Huang X, Yang J (2019) Online pcb defect detector on a new pcb defect dataset. arXiv preprint arXiv:1902.06197

  23. Su BY, Chen HY, Chen P, Bian GB, Liu K, Liu WP (2021) Deep Learning-Based Solar-Cell Manufacturing Defect Detection With Complementary Attention Network. IEEE Trans Ind Inform 17:4084–4095

    Article  Google Scholar 

  24. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. Proc IEEE Conf Comput Vis Pattern Recognit 7132–7141

    Google Scholar 

  25. Tabernik D, Sela S, Skvarc J, Skocaj D (2020) Segmentation-based deep-learning approach for surface-defect detection. J Intell Manuf 31:759–776

    Article  Google Scholar 

  26. Chen JW, Liu ZG, Wang HR, Nunez A, Han ZW (2018) Automatic defect detection of fasteners on the catenary support device using deep convolutional neural network. IEEE Trans Instrum Meas 67:257–269

    Article  Google Scholar 

  27. Mayr M, Hoffmann M, Maier A, Christlein V (2019) Weakly supervised segmentation of cracks on solar cells using normalized L p norm. IEEE Int Conf Image Process (ICIP) 1885–1889

    Google Scholar 

  28. Niu SL, Li B, Wang XG, Lin H (2020) Defect image sample generation with GAN for improving defect recognition. IEEE Trans Autom Sci Eng 17:1611–1622

    Google Scholar 

  29. Huang Q, Wu Y, Baruch J, Jiang P, Peng Y (2009) A template model for defect simulation for evaluating nondestructive testing in x-radiography. IEEE Trans Syst Man Cybern Part A Syst Hum 39:466–475

    Article  Google Scholar 

  30. Lin H, Li B, Wang XG, Shu YF, Niu SL (2019) Automated defect inspection of LED chip using deep convolutional neural network. J Intell Manuf 30:2525–2534

    Article  Google Scholar 

  31. He K, Zhang X, Ren S, Sun J (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 37:1904–1916

    Article  Google Scholar 

  32. Liu S, Qi L, Qin H, Shi J, Jia J (2018) Path aggregation network for instance segmentation. Proc IEEE Conf Comput Vis Pattern Recognit 8759–8768

    Google Scholar 

  33. Wang CY, Liao HYM, Wu YH, Chen PY, Hsieh JW, Yeh IH (2020) CSPNet: A new backbone that can enhance learning capability of CNN. Proc IEEE/CVF Conf Comput Vis Pattern Recognit Workshops 390–391

    Google Scholar 

  34. Zheng Z, Wang P, Liu W, Li J, Ye R, Ren D (2020) Distance-IoU loss: Faster and better learning for bounding box regression. Proc AAAI Conf Artif Intell 34:12993–13000

    Google Scholar 

  35. Grady L (2006) Random walks for image segmentation. IEEE Trans Pattern Anal Mach Intell 28:1768–1783

    Article  Google Scholar 

  36. Lin TY, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for dense object detection. Proc IEEE Int Conf Comput Vis 2980–2988

    Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant No. 61972220) and the Shenzhen Grand Technology Corporation.

Funding

This study was supported by the National Natural Science Foundation of China (Grant No. 61972220) and the Shenzhen Grand Technology Corporation.

Author information

Authors and Affiliations

  1. Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, 518055, People’s Republic of China

    Shuo Wang, Hongyu Wang, Fan Yang & Long Zeng

  2. Shenzhen Grand Technology Corporation, Shenzhen, 518055, People’s Republic of China

    Fei Liu

Authors
  1. Shuo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongyu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fei Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Long Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Wang Shuo, Wang Hongyu and Yang Fan designed and implemented the networks. Liu Fei and Zeng Long contributed to the dataset, network design and writing, and also provided meaningful discussion and support.

Corresponding author

Correspondence to Long Zeng.

Ethics declarations

Ethics approval

The authors declare that this manuscript was not submitted to more than one journal for simultaneous consideration. The submitted work is original and not has been published elsewhere in any form or language.

Consent to participate

The authors declare that they participated in this paper willingly.

Consent for publication

The authors declare to consent to the publication of this paper.

Conflicts of interest/competing interests

The authors declare that they have no conflict of interest.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, S., Wang, H., Yang, F. et al. Attention-based deep learning for chip-surface-defect detection. Int J Adv Manuf Technol 121, 1957–1971 (2022). https://doi.org/10.1007/s00170-022-09425-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-022-09425-4

Keywords

Search

Navigation