Skip to main content
SpringerLink
Log in

Impact of annotation quality on model performance of welding defect detection using deep learning

  • Research Paper
  • Published:
Welding in the World Aims and scope Submit manuscript

Abstract

The use of X-ray-based non-destructive testing (NDT) methods is widespread in the task of welding defect detection. Many scholars have turned to deep-learning computer vision models for defect detection in weld radiographic images in recent years. Before model training, annotating the collected image data is often necessary. We need to use annotation information to guide the model for effective learning. However, many researchers have been focused on developing better models or refining training strategies, often overlooking the quality of data annotation. This paper delved into the impact of eight types of low-quality annotations on the accuracy of object detection models. In comparison to accurate annotations, inaccuracies in the annotated locations significantly impact model performance, while errors in category annotations have a minor effect on model performance. Incorrect location affects both the recall and precision of the model, while incorrect categorization only impacts the precision of the model. Additionally, we observed that the extent of the impact of location errors is related to the detection accuracy of individual classes, with classes having higher original detection AP experiencing more substantial decreases in AP under location errors. Finally, we analyzed the influence of annotator habits on model performance. The study examines the effects of various types of low-quality annotations on model training and their impact on individual detection categories. Annotator habits lead to the left boundary of annotated boxes being less accurate than the right boundary, resulting in a greater impact of annotations biased to the left than those biased to the right. Based on experiments and analysis, we proposed annotation guidelines for weld defect detection tasks: prioritize the quality of location annotations over category accuracy and strive to include all objects, including those with ambiguous boundaries.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data Availability

Data are available under request to corresponding author.

References

  1. Nafaa, N.; Redouane, D.; Amar, B (2000) Weld defect extraction and classification in radiographic testing based artificial neural networks. In Proceedings of the 15th World Conference on Non Destructive Testing, Rome, Italy, p 15–21

  2. Zahran O, Kasban H, El-Kordy M, El-Samie FEA (2013) Automatic weld defect identification from radiographic images. NDT E Int 57:26–35. https://doi.org/10.1016/j.ndteint.2012.11.005

    Article  Google Scholar 

  3. Boaretto N, Centeno TM (2017) Automated detection of welding defects in pipelines from radiographic images DWDI. NDT E Int 86:7–13. https://doi.org/10.1016/j.ndteint.2016.11.003

    Article  CAS  Google Scholar 

  4. Hou W, Wei Y, Jin Y, Zhu C (2019) Deep features based on a DCNN model for classifying imbalanced weld flaw types. Measurement 131:482–489. https://doi.org/10.1016/j.measurement.2018.09.011

    Article  Google Scholar 

  5. Sassi P, Tripicchio P, Avizzano CA (2019) A smart monitoring system for automatic welding defect detection. IEEE Trans Industr Electron 66:9641–9650. https://doi.org/10.1109/TIE.2019.2896165

    Article  Google Scholar 

  6. Zhang Y, You D, Gao X, Zhang N, Gao PP (2019) Welding defects detection based on deep learning with multiple optical sensors during disk laser welding of thick plates. J Manuf Syst 51:87–94. https://doi.org/10.1016/j.jmsy.2019.02.004

    Article  Google Scholar 

  7. Zhang Z, Wen G, Chen S (2019) Weld image deep learning-based on-line defects detection using convolutional neural networks for Al alloy in robotic arc welding. J Manuf Process 45:208–216. https://doi.org/10.1016/j.jmapro.2019.06.023

    Article  Google Scholar 

  8. Yang D, Cui Y, Yu Z, Yuan H (2021) Deep learning based steel pipe weld defect detection. Appl Artif Intell 35:1237–1249. https://doi.org/10.1080/08839514.2021.1975391

    Article  CAS  Google Scholar 

  9. Ji C, Wang H, Li H (2023) Defects detection in weld joints based on visual attention and deep learning. NDT E Int 133:102764. https://doi.org/10.1016/j.ndteint.2022.102764

    Article  Google Scholar 

  10. Wang J, Mu C, Mu S, Zhu R, Yu H (2023) Welding seam detection and location: deep learning network-based approach. Int J Press Vessels Pip 202:104893. https://doi.org/10.1016/j.ijpvp.2023.104893

    Article  Google Scholar 

  11. Cunningham P, Cord M, Delany SJ (2008) Supervised learning. Springer, Berlin Heidelberg

    Book  Google Scholar 

  12. Ma J, Ushiku Y, Sagara M (2022) The effect of improving annotation quality on object detection datasets: a preliminary study. IEEE/CVF Conf Comput Vision Pattern Recog Workshop (CVPRW) 2022:4849–4858. https://doi.org/10.1109/CVPRW56347.2022.00532

    Article  Google Scholar 

  13. Kumar A, Kaur A, Kumar M (2019) Face detection techniques: a review. Artif Intell Rev 52:927–948. https://doi.org/10.1007/s10462-018-9650-2

    Article  Google Scholar 

  14. Mukhtar A, Xia L, Tang TB (2015) Vehicle detection techniques for collision avoidance systems: a review. IEEE Trans Intell Transp Syst 16:2318–2338. https://doi.org/10.1109/TITS.2015.2409109

    Article  Google Scholar 

  15. Zou Z, Chen K, Shi Z, Guo Y, Ye J (2023) Object detection in 20 years: a survey. Proc IEEE 111:257–276. https://doi.org/10.1109/JPROC.2023.3238524

    Article  Google Scholar 

  16. Zhang B, Wang X, Cui J, Wu J, Wang X, Li Y, Li J, Tan Y, Chen X, Wu W, Yu X (2023) Welding defects classification by weakly supervised semantic segmentation. NDT and E Int 138:102899. https://doi.org/10.1016/j.ndteint.2023.102899

    Article  Google Scholar 

  17. Terven J, Córdova-Esparza D-M, Romero-González J-A (2023) A comprehensive review of YOLO architectures in computer vision: from YOLOv1 to YOLOv8 and YOLO-NAS. Mach Learn Knowl Extraction 5:1680–1716. https://doi.org/10.3390/make5040083

    Article  Google Scholar 

  18. Tian Z, Shen C, Chen H, He T (2022) FCOS: a simple and strong anchor-free object detector. IEEE Trans Pattern Anal Mach Intell 44:1922–1933. https://doi.org/10.1109/TPAMI.2020.3032166

    Article  PubMed  Google Scholar 

  19. Shorten C, Khoshgoftaar TM (2019) A survey on image data augmentation for deep learning. Journal of big data 6:1–48. https://doi.org/10.1186/s40537-019-0197-0

    Article  Google Scholar 

  20. Weiss K, Khoshgoftaar TM, Wang D (2016) A survey of transfer learning. J Big data 3:1–40. https://doi.org/10.48550/arXiv.1804.06353

    Article  Google Scholar 

  21. Zhao Z-Q, Zheng P, S-t Xu, Wu X (2019) Object detection with deep learning: a review. IEEE Trans Neural Netw Learn Syst 30:3212–3232. https://doi.org/10.1109/TNNLS.2018.2876865

    Article  PubMed  Google Scholar 

  22. Wang J, Perez L (2017) The effectiveness of data augmentation in image classification using deep learning. Convolutional Neural Netw Vis Recognit 11:1–8. https://doi.org/10.48550/arXiv.1712.04621

    Article  Google Scholar 

  23. Ruby U and V Yendapalli (2020) Binary cross entropy with deep learning technique for image classification. Int J Adv Trends Comput Sci Eng 9: https://doi.org/10.30534/ijatcse/2020/175942020

  24. Li X, Wang W, Hu X, Li J, Tang J, Yang J (2021) Generalized focal loss V2: learning reliable localization quality estimation for dense object detection. IEEE. https://doi.org/10.1109/CVPR46437.2021.01146

    Article  Google Scholar 

  25. Zheng Z, Wang P, Liu W, Li J, Ye R, Ren D (2020) Distance-IoU loss: faster and better learning for bounding box regression. Proc AAAI Conf Artif Intell 34:12993–13000. https://doi.org/10.1609/aaai.v34i07.6999

    Article  Google Scholar 

  26. Neubeck A, Van Gool L (2006) Efficient non-maximum suppression. 18th Int Conf Pattern Recog (ICPR’06) 3:850–855. https://doi.org/10.1109/ICPR.2006.479

    Article  Google Scholar 

  27. Sokolova M, N Japkowicz and S Szpakowicz (2006) Beyond accuracy, F-score and ROC: a family of discriminant measures for performance evaluation. Australas Joint Conf Artif Intell 1015–1021. https://doi.org/10.1007/11941439_114

  28. Selvaraju RR, Das A, Vedantam R, Cogswell M, Parikh D, Batra D (2016) Grad-CAM: Why did you say that? https://doi.org/10.48550/arXiv.1611.07450

Download references

Funding

This work was supported by the Natural Science Foundation of China (NSFC No. U2141216).

Author information

Authors and Affiliations

  1. School of Materials Science & Engineering, Beijing Institute of Technology, 100081, Beijing, People’s Republic of China

    Jinhan Cui, Baoxin Zhang, Xiaopeng Wang, Juntao Wu, Jiajia Liu & Xinghua Yu

  2. Chongqing Innovation Center, Beijing Institute of Technology, 401147, Chongqing, People’s Republic of China

    Yan Li & Xinghua Yu

  3. Chongqing Special Equipment Inspection and Research Institute, 401121, Chongqing, People’s Republic of China

    Xiong Zhi & Wenpin Zhang

Authors
  1. Jinhan Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Baoxin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaopeng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Juntao Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jiajia Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Xiong Zhi
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Wenpin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Xinghua Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xinghua Yu.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Recommended for publication by Commission V—NDT and Quality Assurance of Welded Products.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cui, J., Zhang, B., Wang, X. et al. Impact of annotation quality on model performance of welding defect detection using deep learning. Weld World 68, 855–865 (2024). https://doi.org/10.1007/s40194-024-01710-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40194-024-01710-y

Keywords

Search

Navigation