Abstract
Effective and efficient online quality inspection of poor soldering in Surface Mount Technology (SMT) solder joints, whose defect characteristics are not visually discernible, remains a formidable obstacle in the electronics manufacturing industry, despite the availability of various inspection methods. To overcome this challenge, a novel multimodal fusion method for soldering quality inspection is proposed. First, the method innovatively introduces information from different modalities as input to the inspection model, with the aim to provide more comprehensive information for the decision making of the inspection model. Then, a combination of multimodal gated attention and tensor fusion is used to fuse the features extracted from each modality to form a comprehensive multimodal representation. Finally, this multimodal representation is used to conduct soldering quality inspection. The experimental results demonstrate that the proposed method improves the detection rate of poor soldering significantly, from 93.6 to 99.4%, with a precision level of nearly 100%. The detection rate for visually unapparent soldering defects increases from 49.3 to 95.4%. This meets both manufacturing and customer requirements and to some extent addressing the industry challenge of online SMT soldering quality inspection. This remarkable performance surpasses that of the six mainstream ResNet, ResNext, RegNet, ShuffleNet, EfficientNet and NoisyNet inspection models currently available.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The authors do not have permission to share data.
References
Arevalo, J., Solorio, T., Montes-y-Gómez, M., & González, F. A. (2017). Gated multimodal units for information fusion. arXiv:1702.01992. https://doi.org/10.48550/arXiv.1702.01992.
Cao, S., Parviziomran, I., Park, S., & Won, D. (2019a). Statistical Analysis for Component Shift in pick and place process of Surface Mount Technology. Procedia Manufacturing, 38, 217–224. https://doi.org/10.1016/j.promfg.2020.01.029.
Cao, S., Parviziomran, I., Yang, H., Park, S., & Won, D. (2019b). Prediction of component shifts in pick and place process of surface mount technology using support vector regression. Procedia Manufacturing, 39, 210–217. https://doi.org/10.1016/j.promfg.2020.01.316.
Dai, W., Mujeeb, A., Erdt, M., & Sourin, A. (2020). Soldering defect detection in automatic optical inspection. Advanced Engineering Informatics, 43, 101004. https://doi.org/10.1016/j.aei.2019.101004.
Dorfer, M., Arzt, A., & Widmer, G. (2016). Towards score following in sheet music images. arXiv:1612.05050. https://doi.org/10.48550/arXiv.1612.05050.
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA. https://doi.org/10.1109/CVPR.2016.90.
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735.
Khader, N., & Yoon, S. W. (2021). Adaptive optimal control of stencil printing process using reinforcement learning. Robotics and Computer Integrated Manufacturing, 71, 102132. https://doi.org/10.1016/j.rcim.2021.102132.
Kong, Y., Chae, J., Kim, M., Lee, Y., & Cho, K. (2020). Study on SMT Quality Monitoring by Auto Optical Inspection. 2020 IEEE 22nd Electronics Packaging Technology Conference (EPTC), Singapore, Singapore. https://doi.org/10.1109/EPTC50525.2020.9315137.
Kuo, C. F. J., Fang, T. Y., Lee, C. L., & Wu, H. C. (2019). Automated optical inspection system for surface mount device light emitting diodes. Journal of Intelligent Manufacturing, 30, 641–655. https://doi.org/10.1007/s10845-016-1270-6.
Laurier, C., Grivolla, J., & Herrera, P. (2008). Multimodal music mood classification using audio and lyrics. 2008 seventh international conference on machine learning and applications, San Diego, CA, USA. https://doi.org/10.1109/ICMLA.2008.96.
Li, H., Hao, K., Wei, B., Tang, X. S., & Hu, Q. (2022). A reliable solder joint inspection method based on a light-weight point cloud network and modulated loss. Neurocomputing, 488, 315–327. https://doi.org/10.1016/j.neucom.2022.02.077.
Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Zadeh, A., & Morency, L. P. (2018). Efficient low-rank multimodal fusion with modality-specific factors. arXiv:1806.00064. https://doi.org/10.48550/arXiv.1806.00064.
Liu, T., Bao, J., Wang, J., & Wang, J. (2022). Deep learning for industrial image: Challenges, methods for enriching the sample space and restricting the hypothesis space, and possible issue. International Journal of Computer Integrated Manufacturing, 35(10–11), 1077–1106. https://doi.org/10.1080/0951192X.2021.1901319.
Liu, T., Zheng, P., & Bao, J. (2023). Deep learning-based welding image recognition: A comprehensive review. Journal of Manufacturing Systems, 68, 601–625. https://doi.org/10.1016/j.jmsy.2023.05.026.
Lu, H., Wang, H., Sang, W. Y., & Won, D. (2019). Real-time stencil printing process optimization using a hybrid multi-layer online sequential extreme learning machine and evolutionary search approach. IEEE Transactions on Components Packaging and Manufacturing Technology, 9(12), 2490–2498. https://doi.org/10.1109/TCPMT.2019.2934487.
Mujeeb, A., Dai, W., Erdt, M., & Sourin, A. (2019). One class based feature learning approach for defect detection using deep autoencoders. Advanced Engineering Informatics, 42, 100933. https://doi.org/10.1016/j.aei.2019.100933.
Nazir, Q., & Shao, C. (2021). Online tool condition monitoring for ultrasonic metal welding via sensor fusion and machine learning. Journal of Manufacturing Processes, 62, 806–816. https://doi.org/10.1016/J.JMAPRO.2020.12.050.
Neumayer, R., & Rauber, A. (2007). Integration of text and audio features for genre classification in music information retrieval. Advances in Information Retrieval, ECIR 2007, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-71496-5_78.
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., & Ng, A. Y. (2011). Multimodal deep learning. Proceedings of the 28th International Conference on International Conference on Machine Learning, Bellevue, Washington, USA. https://doi.org/10.5555/3104482.3104569.
Radosavovic, I., Kosaraju, R. P., Girshick, R., He, K., & Dollár, P. (2020). Designing network design spaces. 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Seattle, WA, USA. https://doi.org/10.1109/CVPR42600.2020.01044.
Raihan, F., & Ce, W. (2017). PCB defect detection using OpenCV with image subtraction method. 2017 International Conference on Information Management and Technology (ICIMTech), Yogyakarta, Indonesia. https://doi.org/10.1109/ICIMTech.2017.8273538.
Schindler, A., & Rauber, A. (2015). An audio-visual approach to music genre classification through affective color features. Advances in Information Retrieval, ECIR 2015, Vienna, Austria. https://doi.org/10.1007/978-3-319-16354-3_8.
Srivastava, N., & Salakhutdinov, R. R. (2012). Multimodal learning with deep boltzmann machines. The Journal of Machine Learning Research, 15(1), 2949–2980. https://doi.org/10.5555/2627435.2697059.
Tan, M., & Le, Q. V. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the 36th International Conference on Machine Learning, PMLR. https://doi.org/10.48550/arXiv.1905.11946.
Tsai, T. N. (2008). Modeling and optimization of stencil printing operations: A comparison study. Computers & Industrial Engineering, 54(3), 374–389. https://doi.org/10.1016/j.cie.2007.08.001.
Wang, H., Lu, H., Alelaumi, S. M., & Yoon, S. W. (2021). A wavelet-based multi-dimensional temporal recurrent neural network for stencil printing performance prediction. Robotics and Computer-Integrated Manufacturing, 71, 102129. https://doi.org/10.1016/j.rcim.2021.102129.
Weimer, D., Scholz-Reiter, B., & Shpitalni, M. (2016). Design of deep convolutional neural network architectures for automated feature extraction in industrial inspection. CIRP Annals, 65(1), 417–420. https://doi.org/10.1016/j.cirp.2016.04.072.
Wu, H. (2017). Solder joint defect classification based on ensemble learning. Soldering & Surface Mount Technology, 29(3), 164–170. https://doi.org/10.1108/SSMT-08-2016-0016.
Wu, F., & Zhang, X. (2014). An inspection and classification method for chip solder joints using color grads and boolean rules. Robotics and Computer Integrated Manufacturing, 30(5), 517–526. https://doi.org/10.1016/j.rcim.2014.03.003.
Wu, H., Zhang, X., Xie, H., Kuang, Y., & Ouyang, G. (2013). Classification of solder joint using feature selection based on Bayes and support vector machine. IEEE Transactions on Components Packaging and Manufacturing Technology, 3(3), 516–522. https://doi.org/10.1109/TCPMT.2012.2231902.
Xiao, Z., Wang, Z., Liu, D., & Wang, H. (2022). A path planning algorithm for PCB surface quality automatic inspection. Journal of Intelligent Manufacturing, 33, 1829–1841. https://doi.org/10.1007/s10845-021-01766-3.
Xie, H., Zhang, X., Kuang, Y., & Ouyang, G. (2012). Solder joint inspection method for chip component using improved adaboost and decision tree. IEEE Transactions on Components Packaging & Manufacturing Technology, 1(12), 2018–2027. https://doi.org/10.1109/TCPMT.2011.2168531.
Xie, S., Girshick, R., Dollár, P., Tu, Z., & He, K. (2017). Aggregated residual transformations for deep neural networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA. https://doi.org/10.1109/CVPR.2017.634.
Yu, X., Xue, Y., Zhang, L., Wang, L., Liu, T., & Zhu, D. (2023). NoisyNN: Exploring the Influence of Information Entropy Change in Learning Systems. arXiv:2309.10625. https://doi.org/10.48550/arXiv.2309.10625.
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L. P. (2017). Tensor Fusion Network for Multimodal Sentiment Analysis. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark. https://doi.org/10.18653/v1/D17-1115.
Zhang, K., & Shen, H. (2021). Solder joint defect detection in the connectors using improved faster-rcnn algorithm. Applied Sciences, 11(2), 576. https://doi.org/10.3390/app11020576.
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018). Shufflenet: An extremely efficient convolutional neural network for mobile devices. 2018 IEEE/CVF conference on computer vision and pattern recognition, Salt Lake City, UT, USA. https://doi.org/10.1109/CVPR.2018.00716.
Acknowledgements
The authors express great thanks to the editors and reviewers for their helpful comments and suggestions.
Author information
Authors and Affiliations
Contributions
J.X: Conceptualization, Methodology, Investigation, Formal analysis, Writing – original draft. Y.G: Conceptualization, Funding acquisition, Resources, Supervision, Writing – review & editing. D.L: Conceptualization, Visualization, Writing – review & editing. S.H: Resources, Supervision, Visualization, Writing – review & editing. K.Z: Writing – review & editing. Y.T: Writing – review & editing.
Corresponding authors
Ethics declarations
Competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Xie, J., Guo, Y., Liu, D. et al. A multimodal fusion method for soldering quality online inspection. J Intell Manuf (2024). https://doi.org/10.1007/s10845-024-02413-3
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10845-024-02413-3