Abstract
In laser keyhole welding process, the penetration state is an important index to evaluate the quality of weld seam. In this paper, an innovative deep learning-based monitoring system capable of diagnosing the penetration state in real time is introduced in detail. The built monitoring platform can capture the interaction zone images during the laser welding through a high-speed camera. An adaptive fusion method of adjacent monitoring images is proposed to eliminate the interference of metal vapor plume in the image and highlight the keyhole and molten pool. A deep learning model based on convolutional neural network is established to model the relationship between the interaction zone fusion images and the corresponding penetration states. The constructed training set and validation set can improve the robustness and generalization ability of the built model. The verification results on unfused original images show that the average classification accuracy can reach 98.37%, with a latency about 2.9 ms. Finally, a results-evaluation strategy is proposed to eliminate misclassified results to achieve the state-of-the-art performance. The evaluation results of the entire verification weld seam indicate that the proposed monitoring system can meet the requirements of real-time monitoring with high accuracy and short latency. This study provides a novel and robust method for laser welding penetration state monitoring.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Abbreviations
- CNN:
-
Convolutional neural networks
- BPNN:
-
Back-propagation neural network
- GA:
-
Genetic algorithm
- ANN:
-
Artificial neural network
- ANFIS:
-
Adaptive neuro fuzzy inference system
- SCNN:
-
Spectral convolutional neural network
- ROI:
-
Region of interest
- CPU:
-
Central processing unit
- GPU:
-
Graphics processing unit
- PS I, II, III, IV:
-
Penetration state I, II, III, IV
- BN:
-
Batch normalization
- ReLU:
-
Rectified linear unit
References
Abderrazak, K., Ben, S. W., Mhiri, H., Bournot, P., & Autric, M. (2009). Nd:YAG laser welding of AZ91 magnesium alloy for aerospace industries. Metallurgical and Materials Transactions B, 40(1), 54–61. https://doi.org/10.1007/s11663-008-9218-7
Ai, Y. W., Jiang, P., Wang, C. M., Mi, G. Y., & Geng, S. N. (2018). Experimental and numerical analysis of molten pool and keyhole profile during high-power deep-penetration laser welding. International Journal of Heat and Mass Transfer, 126, 779–789. https://doi.org/10.1016/j.ijheatmasstransfer.2018.05.031
Arnold, G. (2009). Laser micro manufacturing: Fast and reliable solutions for joining, drilling and structuring. Laser Technik Journal, 6(1), 16–19. https://doi.org/10.1002/latj.200990001
Cai, W., Wang, J. Z., Jiang, P., Cao, L. C., Mi, G. Y., & Zhou, Q. (2020). Application of sensing techniques and artificial intelligence-based methods to laser welding real-time monitoring: A critical review of recent literature. Journal of Manufacturing Systems, 57, 1–18. https://doi.org/10.1016/j.jmsy.2020.07.021
Cao, X., Wanjara, P., Huang, J., Munro, C., & Nolting, A. (2011). Hybrid fiber laser—Arc welding of thick section high strength low alloy steel. Materials and Design, 32(6), 3399–3413. https://doi.org/10.1016/j.matdes.2011.02.002
Chandrasekhar, N., Vasudevan, M., Bhaduri, A. K., & Jayakumar, T. (2013). Intelligent modeling for estimating weld bead width and depth of penetration from infra-red thermal images of the weld pool. Journal of Intelligent Manufacturing, 26(1), 59–71. https://doi.org/10.1007/s10845-013-0762-x
Dubey, A. K., & Yadava, V. (2008). Laser beam machining—A review. International Journal of Machine Tools and Manufacture, 48(6), 609–628. https://doi.org/10.1016/j.ijmachtools.2007.10.017
García-Moreno, A. I. (2021). A fast method for monitoring molten pool in infrared image streams using gravitational superpixels. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-021-01761-8
Gonzalez-Val, C., Pallas, A., Panadeiro, V., & Rodriguez, A. (2019). A convolutional approach to quality monitoring for laser manufacturing. Journal of Intelligent Manufacturing, 31(3), 789–795. https://doi.org/10.1007/s10845-019-01495-8
Gu, H., Väistö, T., & Li, L. (2020). Numerical and experimental study on the molten pool dynamics and fusion zone formation in multi-pass narrow gap laser welding. Optics and Laser Technology, 126, 106081. https://doi.org/10.1016/j.optlastec.2020.106081
Hoang, D. T., & Kang, H. J. (2019). A survey on Deep Learning based bearing fault diagnosis. Neurocomputing, 335, 327–335. https://doi.org/10.1016/j.neucom.2018.06.078
Hong, K. M., & Shin, Y. C. (2017). Prospects of laser welding technology in the automotive industry: A review. Journal of Materials Processing Technology, 245, 46–69. https://doi.org/10.1016/j.jmatprotec.2017.02.008
Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning, 37, 448–456. https://doi.org/10.5555/3045118.3045167
Jarrett, K., Kavukcuoglu, K., Ranzato, M., Lecun, Y., & IEEE (2009). What is the best multi-stage architecture for object recognition? In 2009 IEEE 12th International Conference on Computer Vision (pp. 2146–2153). IEEE).
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., et al. (1990). Handwritten digit recognition with a back-propagation network. Advances in Neural Information Processing Systems, 2, 396–404.
Lei, Z. L., Shen, J. X., Wang, Q., & Chen, Y. B. (2019). Real-time weld geometry prediction based on multi-information using neural network optimized by PCA and GA during thin-plate laser welding. Journal of Manufacturing Processes, 43, 207–217. https://doi.org/10.1016/j.jmapro.2019.05.013
Liu, G. Q., Gao, X. D., You, D. Y., & Zhang, N. F. (2016). Prediction of high power laser welding status based on PCA and SVM classification of multiple sensors. Journal of Intelligent Manufacturing, 30(2), 821–832. https://doi.org/10.1007/s10845-016-1286-y
Liu, Y., & Zhang, Y. (2015). Iterative local ANFIS-based human welder intelligence modeling and control in pipe GTAW process: A data-driven approach. IEEE/ASME Transactions on Mechatronics, 20(3), 1079–1088. https://doi.org/10.1109/tmech.2014.2363050
Luo, M., & Shin, Y. C. (2015). Vision-based weld pool boundary extraction and width measurement during keyhole fiber laser welding. Optics and Lasers in Engineering, 64, 59–70. https://doi.org/10.1016/j.optlaseng.2014.07.004
Meng, W., Yin, X. H., Fang, J. F., Guo, L. J., Ma, Q. S., & Li, Z. G. (2019). Dynamic features of plasma plume and molten pool in laser lap welding based on image monitoring and processing techniques. Optics and Laser Technology, 109, 168–177. https://doi.org/10.1016/j.optlastec.2018.07.073
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., et al. (2019). PyTorch: An imperative style, high-performance deep learning library. arXiv:1912.01703
Shevchik, S., Kenel, C., Leinenbach, C., & Wasmer, K. (2018). Acoustic emission for in situ quality monitoring in additive manufacturing using spectral convolutional neural networks. Additive Manufacturing, 21, 598–604. https://doi.org/10.1016/j.addma.2017.11.012
Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. Computer Vision and Pattern Recognition. arXiv:1409.1556v2
Wang, Q. Y., Jiao, W. H., & Zhang, Y. M. (2020). Deep learning-empowered digital twin for visualized weld joint growth monitoring and penetration control. Journal of Manufacturing Systems, 57, 429–439. https://doi.org/10.1016/j.jmsy.2020.10.002
Wang, X. W., & Li, R. R. (2013). Intelligent modelling of back-side weld bead geometry using weld pool surface characteristic parameters. Journal of Intelligent Manufacturing, 25(6), 1301–1313. https://doi.org/10.1007/s10845-013-0731-4
Xiong, J., & Zou, S. Y. (2019). Active vision sensing and feedback control of back penetration for thin sheet aluminum alloy in pulsed MIG suspension welding. Journal of Process Control, 77, 89–96. https://doi.org/10.1016/j.jprocont.2019.03.013
Xu, Z. Y., Wang, R., Zou, R., & Yu, L. D. Prediction of weld penetration status based on sparse representation in fiber laser welding. In Ninth international symposium on precision mechanical measurements, Chongqing, 2019 (Vol. 11343, p. 106): Proceedings of SPIE. https://doi.org/10.1117/12.2548813.
Zhang, W. J., Zhang, X., & Zhang, Y. M. (2013). Robust pattern recognition for measurement of three dimensional weld pool surface in GTAW. Journal of Intelligent Manufacturing, 26(4), 659–676. https://doi.org/10.1007/s10845-013-0825-z
Zhang, Y., Li, F. Z., Liang, Z. C., Ying, Y. Y., Lin, Q. D., & Wei, H. Y. (2018). Correlation analysis of penetration based on keyhole and plasma plume in laser welding. Journal of Materials Processing Technology, 256, 1–12. https://doi.org/10.1016/j.jmatprotec.2018.01.032
Zhang, Y., Liu, T. W., Li, B., & Zhang, Z. H. (2019a). Simultaneous monitoring of penetration status and joint tracking during laser keyhole welding. IEEE/ASME Transactions on Mechatronics, 24(4), 1732–1742. https://doi.org/10.1109/tmech.2019.2916984
Zhang, Y. X., You, D. Y., Gao, X. D., Wang, C. Y., Li, Y. J., & Gao, P. P. (2019b). Real-time monitoring of high-power disk laser welding statuses based on deep learning framework. Journal of Intelligent Manufacturing, 31(4), 799–814. https://doi.org/10.1007/s10845-019-01477-w
Zhang, Y. X., You, D. Y., Gao, X. D., Zhang, N. F., & Gao, P. P. (2019c). Welding defects detection based on deep learning with multiple optical sensors during disk laser welding of thick plates. Journal of Manufacturing Systems, 51, 87–94. https://doi.org/10.1016/j.jmsy.2019.02.004
Zhang, Z. F., & Chen, S. B. (2014). Real-time seam penetration identification in arc welding based on fusion of sound, voltage and spectrum signals. Journal of Intelligent Manufacturing, 28(1), 207–218. https://doi.org/10.1007/s10845-014-0971-y
Zhang, Z. H., Li, B., Zhang, W. F., Lu, R. D., Wada, S., & Zhang, Y. (2020). Real-time penetration state monitoring using convolutional neural network for laser welding of tailor rolled blanks. Journal of Manufacturing Systems, 54, 348–360. https://doi.org/10.1016/j.jmsy.2020.01.006
Zhang, Z. F., Wen, G. R., & Chen, S. B. (2019d). Weld image deep learning-based on-line defects detection using convolutional neural networks for Al alloy in robotic arc welding. Journal of Manufacturing Processes, 45, 208–216. https://doi.org/10.1016/j.jmapro.2019.06.023
Zou, S. Y., Wang, Z. J., Hu, S. S., Wang, W. D., & Cao, Y. (2019). Control of weld penetration depth using relative fluctuation coefficient as feedback. Journal of Intelligent Manufacturing, 31(5), 1203–1213. https://doi.org/10.1007/s10845-019-01506-8
Acknowledgements
This research has been supported by the National Natural Science Foundation of China under Grant Nos. 52075201, 51861165202, the Postdoctoral Science Foundation of China under Grant No. 2020M682407, the opening project of State Key Laboratory of Digital Manufacturing Equipment and Technology (HUST) under Grant No. DMETKF2018001, and the Fundamental Research Funds for the Central Universities, HUST: 2019JYCXJJ024.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Cai, W., Jiang, P., Shu, L. et al. Real-time laser keyhole welding penetration state monitoring based on adaptive fusion images using convolutional neural networks. J Intell Manuf 34, 1259–1273 (2023). https://doi.org/10.1007/s10845-021-01848-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10845-021-01848-2