Skip to main content
SpringerLink
Log in

Learning semantic-specific visual representation for laser welding penetration status recognition

  • Article
  • Published:
Science China Technological Sciences Aims and scope Submit manuscript

Abstract

The degree of penetration can directly reflect the forming quality of laser welding. The fine-grained feature of the molten pool/keyhole image brings challenges to the vision-based laser welding penetration status recognition. In this paper, a novel knowledge-and-data-hybrid driven recognition model is proposed for solving the problem of difficult learning of discriminative visual features of molten pool/keyhole images. In addition, a label semantic attention mechanism (LSA) is designed with three modules: representation of image visual feature, representation of labels semantic feature, and generation of label semantic attention. For learning discriminative features in visual space, LSA uses discriminative information in label semantics to guide the convolutional neural network. The experimental results show that the proposed LSA method has faster convergence and higher accuracy than the traditional attention mechanism. Further comparative experiments reveal that LSA is less dependent on the amount of training data and model complexity. The results of visualization experiments show that the visual features learned by the proposed method are more discriminative.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Rong Y, Mi G, Xu J, et al. Laser penetration welding of ship steel EH36: A new heat source and application to predict residual stress considering martensite phase transformation. Mar Struct, 2018, 61: 256–267

    Article  Google Scholar 

  2. Zhang S, Wang Y, Zhu M, et al. Effects of heat source arrangements on laser-MAG hybrid welding characteristics and defect formation mechanism of 10CrNi3MoV steel. J Manuf Process, 2020, 58: 563–573

    Article  Google Scholar 

  3. Chen Y, Chen B, Yao Y, et al. A spectroscopic method based on support vector machine and artificial neural network for fiber laser welding defects detection and classification. NDT E Int, 2019, 108: 102176

    Article  Google Scholar 

  4. Zhang R, Tang X, Xu L, et al. Mechanism study of thermal fluid flow and weld root hump suppression in full penetration laser welding of Al alloy with alternating magnetic field support. Int J Heat Mass Transfer, 2021, 166: 120759

    Article  Google Scholar 

  5. Fetzer F, Hagenlocher C, Weber R, et al. Geometry and stability of the capillary during deep-penetration laser welding of AlMgSi at high feed rates. Opt Laser Tech, 2021, 133: 106562

    Article  Google Scholar 

  6. Zhang Y, Li F, Liang Z, et al. Correlation analysis of penetration based on keyhole and plasma plume in laser welding. J Mater Process Tech, 2018, 256: 1–12

    Article  Google Scholar 

  7. Song S, Chen H, Lin T, et al. Penetration state recognition based on the double-sound-sources characteristic of VPPAW and hidden Markov Model. J Mater Process Tech, 2016, 234: 33–44

    Article  Google Scholar 

  8. Yusof M F M, Ishak M, Ghazali M F. Classification of weld penetration condition through synchrosqueezed-wavelet analysis of sound signal acquired from pulse mode laser welding process. J Mater Process Tech, 2020, 279: 116559

    Article  Google Scholar 

  9. Zhang S, Hu S, Wang Z. Weld penetration sensing in pulsed gas tungsten arc welding based on arc voltage. J Mater Process Tech, 2016, 229: 520–527

    Article  Google Scholar 

  10. Huang Y, Li S, Li J, et al. Spectral diagnosis and defects prediction based on ELM during the GTAW of Al alloys. Measurement, 2019, 136: 405–414

    Article  Google Scholar 

  11. Huang Y, Wu D, Zhang Z, et al. EMD-based pulsed TIG welding process porosity defect detection and defect diagnosis using GASVM. J Mater Process Tech, 2017, 239: 92–102

    Article  Google Scholar 

  12. Zhang Y, Zhang N, You D, et al. High-power disk laser welding statuses monitoring based on analyses of multiple-sensor signals. J Manuf Process, 2019, 41: 221–230

    Article  Google Scholar 

  13. Lu R, Wei H, Li F, et al. In-situ monitoring of the penetration status of keyhole laser welding by using a support vector machine with interaction time conditioned keyhole behaviors. Opt Laser Eng, 2020, 130: 106099

    Article  Google Scholar 

  14. Wu D, Chen H, Huang Y, et al. Monitoring of weld joint penetration during variable polarity plasma arc welding based on the keyhole characteristics and PSO-ANFIS. J Mater Process Tech, 2017, 239: 113–124

    Article  Google Scholar 

  15. Sheng J, Cai Y, Li F, et al. Online detection method of weld penetration based on molten pool morphology and metallic vapor radiation for fiber laser welding. Int J Adv Manuf Technol, 2017, 92: 231–245

    Article  Google Scholar 

  16. Tang Z, Liu W, Zhang N, et al. Real-time prediction of penetration depths of laser surface melting based on coaxial visual monitoring. Opt Laser Eng, 2020, 128: 106034

    Article  Google Scholar 

  17. Cai W, Wang J Z, Jiang P, et al. Application of sensing techniques and artificial intelligence-based methods to laser welding real-time monitoring: A critical review of recent literature. J Manuf Syst, 2020, 57: 1–18

    Article  Google Scholar 

  18. Wang B, Hu S J, Sun L, et al. Intelligent welding system technologies: State-of-the-art review and perspectives. J Manuf Syst, 2020, 56: 373–391

    Article  Google Scholar 

  19. Wang Q, Jiao W, Zhang Y M. Deep learning-empowered digital twin for visualized weld joint growth monitoring and penetration control. J Manuf Syst, 2020, 57: 429–439

    Article  Google Scholar 

  20. Zhang B, Zhu J, Su H. Toward the third generation of artificial intelligence (in Chinese). Sci Sin Inform, 2020, 50: 1281–1302

    Article  Google Scholar 

  21. Huang S Y, Su H, Zhu J, et al. Combo-action: Training agent for FPS game with auxiliary tasks. In: The Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Honolulu, 2019. 954–961

  22. Senderovich A, Francescomarino C D, Maggi F M. From knowledge-driven to data-driven inter-case feature encoding in predictive process monitoring. Inf Syst, 2019, 84: 255–264

    Article  Google Scholar 

  23. Liu X, Zhou Y, Zhao H. Robust hierarchical feature selection driven by data and knowledge. Inf Sci, 2021, 551: 341–357

    Article  MathSciNet  Google Scholar 

  24. Kou L, Liu C, Cai G, et al. Fault diagnosis for open-circuit faults in NPC inverter based on knowledge-driven and data-driven approaches. IET Power Electron, 2020, 13: 1236–1245

    Article  Google Scholar 

  25. Fan X R, Wang X, Kang M, et al. A knowledge-and-data-driven modeling approach for simulating plant growth and the dynamics of CO2/O2 concentrations in a closed system of plants and humans by integrating mechanistic and empirical models. Comput Electron Agr, 2018, 148: 280–290

    Article  Google Scholar 

  26. Zhang Y, You D, Gao X, et al. Welding defects detection based on deep learning with multiple optical sensors during disk laser welding of thick plates. J Manuf Syst, 2019, 51: 87–94

    Article  Google Scholar 

  27. He K, Zhang X, Ren S, et al. Deep residual learning for image recognition. In: The Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, 2016

  28. Ren S, He K, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell, 2017, 39: 1137–1149

    Article  Google Scholar 

  29. Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. In: The Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Boston, 2015

  30. Li C, Wang Q, Jiao W, et al. Deep learning-based detection of penetration from weld pool reflection images. Weld J, 2020, 99: 239s–245s

    Article  Google Scholar 

  31. Wu D, Huang Y, Zhang P, et al. Visual-acoustic penetration recognition in variable polarity plasma arc welding process using hybrid deep learning approach. IEEE Access, 2020, 8: 120417

    Article  Google Scholar 

  32. Jiao W, Wang Q, Cheng Y, et al. End-to-end prediction of weld penetration: A deep learning and transfer learning based method. J Manuf Process, 2021, 63: 191–197

    Article  Google Scholar 

  33. Zhang Z, Li B, Zhang W, et al. Real-time penetration state monitoring using convolutional neural network for laser welding of tailor rolled blanks. J Manuf Syst, 2020, 54: 348–360

    Article  Google Scholar 

  34. Yang Y, Yang R, Pan L, et al. A lightweight deep learning algorithm for inspection of laser welding defects on safety vent of power battery. Comput Ind, 2020, 123: 103306

    Article  Google Scholar 

  35. Pan H, Pang Z, Wang Y, et al. A new image recognition and classification method combining transfer learning algorithm and mobilenet model for welding defects. IEEE Access, 2020, 8: 119951

    Article  Google Scholar 

  36. Liu T, Bao J, Wang J, et al. A coarse-grained regularization method of convolutional kernel for molten pool defect identification. J Comput Inf Sci Eng, 2020, 20: 021005

    Article  Google Scholar 

  37. Xia C, Pan Z, Fei Z, et al. Vision based defects detection for keyhole TIG welding using deep learning with visual explanation. J Manuf Process, 2020, 56: 845–855

    Article  Google Scholar 

  38. Cui Y, Song Y, Sun C, et al. Large scale fine-grained categorization and domain-specific transfer learning. In: The Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, 2018

  39. Zhang N, Donahue J, Girshick R B, et al. Part-based R-CNNs for fine-grained category detection. In: The Proceedings of European Conference on Computer Vision. Zurich, 2014

  40. Branson S, Beijbom O, Belongie S. Efficient large-scale structured learning. In: The Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Portland, 2013

  41. Wei X S, Xie C W, Wu J. Mask-CNN: Localizing parts and selecting descriptors for fine-grained image recognition. arXiv: 1605.06878, 2016

  42. Fu J L, Zheng H L, Mei T. Look Closer to See Better: Recurrent attention convolutional neural network for fine-grained image recognition. In: The Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, 2017

  43. Xiao T J, Xu Y C, Yang K Y, et al. The application of two-level attention models in deep convolutional neural network for fine-grained image classification. In: The Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Boston, 2015

  44. Sun M, Yuan Y, Zhou F, et al. Multi-attention multi-class constraint for fine-grained image. In: The Proceedings of European Conference on Computer Vision. Munich, 2018

  45. Jaderberg M, Simonyan K. Spatial transformer networks. arXiv: 1506.02025, 2015

  46. Mnih V, Heess N, Graves A, et al. Recurrent models of visual attention. In: The Proceedings of Conference on Neural Information Processing Systems. Montreal, 2014

  47. Hu J, Shen L, Albanie S, et al. Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell, 2020, 42: 2011–2023

    Article  Google Scholar 

  48. Li X, Wang W H, Hu X L, et al. Selective kernel networks. In: The Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Long Beach, 2019

  49. Li H C, Xiong P F, An J, et al. Pyramid attention network for semantic segmentation. arXiv: 1805.10180, 2018

  50. Woo S, Park J, Lee J Y, et al. CBAM: Convolutional block attention module. arXiv: 1807.06521, 2018

  51. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv: 1409.1556, 2015

  52. Pennington J, Socher R, Manning C. Glove: Global vectors for word representation. In: The Proceedings of Conference on Empirical Methods in Natural Language Processing. Doha, 2014

  53. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput, 1997, 9: 1735–1780

    Article  Google Scholar 

  54. Dumais S T. Latent semantic analysis. Ann Rev Inf Sci Tech, 2005, 38: 188–230

    Article  Google Scholar 

  55. Le Q V, Mikolov T. Distributed representations of sentences and documents. arXiv: 1405.4053, 2014

  56. Zhou Z H. Machine Learning (in Chinese). Beijing: Tsinghua University Press, 2016. 267–287

    Google Scholar 

  57. Lecun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition. Proc IEEE, 1998, 86: 2278–2324

    Article  Google Scholar 

  58. Selvaraju R R, Cogswell M, Das A, et al. Grad-CAM: Visual explanations from deep networks via gradient-based localization. Int J Comput Vis, 2020, 128: 336–359

    Article  Google Scholar 

  59. Maaten L V D, Hinton G. Visualizing data using t-SNE. J Mach Learn Res, 2008, 9: 2579–2605

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Intelligent Manufacturing, College of Mechanical Engineering, Donghua University, Shanghai, 201600, China

    TianYuan Liu, JinSong Bao, HangBin Zheng & JunLiang Wang

  2. Shanghai Spaceflight Precision Machinery Institute, Shanghai, 201600, China

    ChangQi Yang

  3. Shanghai Institute of Laser Technology, Shanghai, 200235, China

    Jun Gu

Authors
  1. TianYuan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. JinSong Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. HangBin Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. JunLiang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. ChangQi Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jun Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to JinSong Bao.

Additional information

This work was supported by the Fundamental Research Funds for the Central Universities and Graduate Student Innovation Fund of Donghua University (Grant No. CUSF-DH-D-2020053).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, T., Bao, J., Zheng, H. et al. Learning semantic-specific visual representation for laser welding penetration status recognition. Sci. China Technol. Sci. 65, 347–360 (2022). https://doi.org/10.1007/s11431-021-1848-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11431-021-1848-7

Keywords

Search

Navigation