Abstract
With the rapid development of additive manufacturing (AM) technology, quality inspection has become one of the most crucial research topics in additive manufacturing. Although numerous image-based deep learning methods have been successfully developed to monitor and inspect AM product quality effectively, many require substantial labels in order to achieve satisfactory training, which is often impractical in real-life AM processes. In this article, a novel image feature-based self-supervised learning (IFSSL) model is proposed for effective quality inspection in AM. Through a feature-based image fusion approach based on defect-relevant feature extraction, the IFSSL model is able to guide machine vision to focus on highlighted defect-relevant regions in the AM product image. In addition, the defect-relevant features are used to generate pseudo-labels for self-supervised learning. With self-supervision, the IFSSL model leverages the advantages of supervised learning and unsupervised learning by requiring no sample label while retaining defect-relevant information. The effectiveness of the proposed IFSSL method is demonstrated through a real case study of fused deposition modeling product image dataset. Results show that the IFSSL model can guide machine vision to pay more attention to potential defective regions, enabling it to detect and locate faults effectively and automatically for machine vision guided quality inspection.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The Fused Deposition Modeling (FDM) 3D Printing Specimens Scan dataset used in this study are available to the public under a Creative Commons Attribution (CC-BY) license at https://doi.org/10.5281/zenodo.159676. Description of the dataset can be found in the open access article by Baumann et al. (2017) at https://doi.org/10.3390/data2010003.
References
Agarwal, A., Samaiya, D., & Gupta, K. K. (2017, December). A comparative study of SIFT and SURF algorithms under different object and background conditions. In 2017 international conference on information technology (ICIT) (pp. 42-45). IEEE. https://doi.org/10.1109/ICIT.2017.48
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021). Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions. Journal of Big Data,8(1), 53. https://doi.org/10.1186/s40537-021-00444-8
Arsomngern, P., Long, C., Suwajanakorn, S., & Nutanong, S. (2023). Towards pointsets representation learning via self-supervised learning and set augmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence,45(1), 1201–1216. https://doi.org/10.1109/TPAMI.2021.3139113
Baumann, F. W., Eichhoff, J. R., & Roller, D. (2017). Scanned image data from 3D-printed specimens using fused deposition modeling. Data. https://doi.org/10.3390/data2010003
Bay, H., Ess, A., Tuytelaars, T., & Van Gool, L. (2008). Speeded-up robust features (SURF). Computer Vision and Image Understanding,110(3), 346–359. https://doi.org/10.1016/j.cviu.2007.09.014
BT.601, I.-R. R. (2011). Studio encoding parameters of digital television for standard 4: 3 and wide-screen 16: 9 aspect ratios. International radio consultative committee international telecommunication union, Switzerland, CCIR Rep.
Bugatti, M., & Colosimo, B. M. (2022). Towards real-time in-situ monitoring of hot-spot defects in L-PBF: A new classification-based method for fast video-imaging data analysis. Journal of Intelligent Manufacturing,33(1), 293–309. https://doi.org/10.1007/s10845-021-01787-y
Colosimo, B. M., & Grasso, M. (2018). Spatially weighted PCA for monitoring video image data with application to additive manufacturing. Journal of Quality Technology,50(4), 391–417. https://doi.org/10.1080/00224065.2018.1507563
Colosimo, B. M., Huang, Q., Dasgupta, T., & Tsung, F. (2018). Opportunities and challenges of quality engineering for additive manufacturing. Journal of Quality Technology,50(3), 233–252. https://doi.org/10.1080/00224065.2018.1487726
Denlinger, E. R., Heigel, J. C., Michaleris, P., & Palmer, T. A. (2015). Effect of inter-layer dwell time on distortion and residual stress in additive manufacturing of titanium and nickel alloys. Journal of Materials Processing Technology,215, 123–131. https://doi.org/10.1016/j.jmatprotec.2014.07.030
Donoser, M., & Bischof, H. (2006). Efficient maximally stable extremal region (MSER) tracking. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06).https://doi.org/10.1109/CVPR.2006.107
Efros, A. A., & Leung, T. K. (1999). Texture synthesis by non-parametric sampling. In Proceedings of the seventh IEEE international conference on computer vision (ICCV), https://doi.org/10.1109/iccv.1999.790383.
Ericsson, L., Gouk, H., & Hospedales, T. M. (2021). How well do self-supervised models transfer? In 2021 IEEE/CVF conference on computer vision and pattern recognition (CVPR), https://doi.org/10.1109/cvpr46437.2021.00537.
Ericsson, L., Gouk, H., Loy, C. C., & Hospedales, T. M. (2022). Self-supervised representation learning: Introduction, advances, and challenges. IEEE Signal Processing Magazine,39(3), 42–62. https://doi.org/10.1109/msp.2021.3134634
Forssén, PE. (2007). Maximally stable colour regions for recognition and matching. In 2007 IEEE conference on computer vision and recognition P (CVPR’07). https://doi.org/10.1109/CVPR.2007.383120.
Gao, X. W., Taylor, S., Pang, W., Hui, R., Lu, X., & Braden, B. (2023). Fusion of colour contrasted images for early detection of oesophageal squamous cell dysplasia from endoscopic videos in real time. Information Fusion,92, 64–79. https://doi.org/10.1016/j.inffus.2022.11.023
Gao, Y., Li, X., Wang, X. V., Wang, L., & Gao, L. (2022). A review on recent advances in vision-based defect recognition towards industrial intelligence. Journal of Manufacturing Systems,62, 753–766. https://doi.org/10.1016/j.jmsy.2021.05.008
Guo, S., Agarwal, M., Cooper, C., Tian, Q., Gao, R. X., Guo, W., & Guo, Y. B. (2022). Machine learning for metal additive manufacturing: towards a physics-informed data-driven paradigm. Journal of Manufacturing Systems,62, 145–163. https://doi.org/10.1016/j.jmsy.2021.11.003
Hao, R., Lu, B., Cheng, Y., Li, X., & Huang, B. (2020). A steel surface defect inspection approach towards smart industrial monitoring. Journal of Intelligent Manufacturing,32(7), 1833–1843. https://doi.org/10.1007/s10845-020-01670-2
Harris, C., & Stephens, M. (1988). A combined corner and edge detector. In Alvey vision conference. https://doi.org/10.5244/c.2.23.
Herzog, T., Brandt, M., Trinchi, A., Sola, A., & Molotnikov, A. (2023). Process monitoring and machine learning for defect detection in laser-based metal additive manufacturing. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-023-02119-y
Huang, C., Yang, Z., Wen, J., Xu, Y., Jiang, Q., Yang, J., & Wang, Y. (2022). Self-Supervision-augmented deep autoencoder for unsupervised visual anomaly detection. IEEE Transactions on Cybernetics,52(12), 13834–13847. https://doi.org/10.1109/TCYB.2021.3127716
Jing, L., & Tian, Y. (2021). Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence,43(11), 4037–4058. https://doi.org/10.1109/TPAMI.2020.2992393
Kim, H., Lee, H., & Ahn, S. H. (2022). Systematic deep transfer learning method based on a small image dataset for spaghetti-shape defect monitoring of fused deposition modeling. Journal of Manufacturing Systems,65, 439–451. https://doi.org/10.1016/j.jmsy.2022.10.009
Kim, H., Lin, Y., & Tseng, T. L. B. (2018). A review on quality control in additive manufacturing. Rapid Prototyping Journal,24(3), 645–669. https://doi.org/10.1108/rpj-03-2017-0048
Kumar, S., Gopi, T., Harikeerthana, N., Gupta, M. K., Gaur, V., Krolczyk, G. M., & Wu, C. (2023). Machine learning techniques in additive manufacturing: A state of the art review on design, processes and production control. Journal of Intelligent Manufacturing,34(1), 21–55. https://doi.org/10.1007/s10845-022-02029-5
Lei, X., Zhao, G., & Kuo, C. C. J. (2020). NITES: A non-parametric interpretable texture synthesis method. In 2020 Asia-Pacific signal and information processing association annual summit and conference (APSIPA ASC) (pp. 1698–1706). IEEE.
Li, J., Chen, E., Ding, Z., Zhu, L., Lu, K., & Shen, H. T. (2021). Maximum density divergence for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence,43(11), 3918–3930. https://doi.org/10.1109/TPAMI.2020.2991050
Li, J., Zhou, Q., Huang, X., Li, M., & Cao, L. (2023). In situ quality inspection with layer-wise visual images based on deep transfer learning during selective laser melting. Journal of Intelligent Manufacturing,34(2), 853–867. https://doi.org/10.1007/s10845-021-01829-5
Li, X., Jia, X., Yang, Q., & Lee, J. (2020). Quality analysis in metal additive manufacturing with deep learning. Journal of Intelligent Manufacturing,31(8), 2003–2017. https://doi.org/10.1007/s10845-020-01549-2
Liu, C., Tian, W., & Kan, C. (2022). When AI meets additive manufacturing: Challenges and emerging opportunities for human-centered products development. Journal of Manufacturing Systems,64, 648–656. https://doi.org/10.1016/j.jmsy.2022.04.010
Liu, C., Wang, K., Wang, Y., & Yuan, X. (2022). Learning deep multimanifold structure feature representation for quality prediction with an industrial application. IEEE Transactions on Industrial Informatics,18(9), 5849–5858. https://doi.org/10.1109/tii.2021.3130411
Liu, D., Wang, Y., Liu, C., Yuan, X., Yang, C., & Gui, W. (2023). Data mode related interpretable transformer network for predictive modeling and key sample analysis in industrial processes. IEEE Transactions on Industrial Informatics,19(9), 9325–9336. https://doi.org/10.1109/tii.2022.3227731
Liu, Z., Song, Y., Tang, R., Duan, G., & Tan, J. (2023). Few-shot defect recognition of metal surfaces via attention-embedding and self-supervised learning. Journal of Intelligent Manufacturing,34(8), 3507–3521. https://doi.org/10.1007/s10845-022-02022-y
Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision,60, 91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94
Lui, C. F., Liu, Y., & Xie, M. (2022). A supervised bidirectional long short-term memory network for data-driven dynamic soft sensor modeling. IEEE Transactions on Instrumentation and Measurement,71, 1–13. https://doi.org/10.1109/tim.2022.3152856
Maged, A., Lui, C. F., Haridy, S., & Xie, M. (2023). Variational AutoEncoders-LSTM based fault detection of time-dependent high dimensional processes. International Journal of Production Research,1–16, 1. https://doi.org/10.1080/00207543.2023.2175591
Maged, A., & Xie, M. (2022). Uncertainty utilization in fault detection using bayesian deep learning. Journal of Manufacturing Systems,64, 316–329. https://doi.org/10.1016/j.jmsy.2022.07.002
Matas, J., Chum, O., Urban, M., & Pajdla, T. (2004). Robust wide-baseline stereo from maximally stable extremal regions. Image and Vision Computing,22(10), 761–767. https://doi.org/10.1016/j.imavis.2004.02.006
Miao, Y., Jeon, J. Y., & Park, G. (2020). An image processing-based crack detection technique for pressed panel products. Journal of Manufacturing Systems,57, 287–297. https://doi.org/10.1016/j.jmsy.2020.10.004
Mukherjee, T., Zuback, J. S., De, A., & DebRoy, T. (2016). Printability of alloys for additive manufacturing. Scientific Reports,6, 19717. https://doi.org/10.1038/srep19717
Ngo, T. D., Kashani, A., Imbalzano, G., Nguyen, K. T. Q., & Hui, D. (2018). Additive manufacturing (3D printing): A review of materials, methods, applications and challenges. Composites Part B: Engineering,143, 172–196. https://doi.org/10.1016/j.compositesb.2018.02.012
Nistér, D., & Stewénius, H. (2008). Linear time maximally stable extremal regions. In D. Forsyth, P. Torr, & A. Zisserman, Computer vision—ECCV 2008 European Conference on Computer Vision (ECCV) 2008, https://doi.org/10.1007/978-3-540-88688-4_14
Piao, Z., Tang, L., & Zhao, B. (2023). Unsupervised domain-adaptive object detection via localization regression alignment. IEEE Transactions on Neural Networks and Learning Systems,1–12, 1. https://doi.org/10.1109/TNNLS.2023.3282958
Qin, J., Hu, F., Liu, Y., Witherell, P., Wang, C. C. L., Rosen, D. W., Simpson, T. W., Lu, Y., & Tang, Q. (2022). Research and application of machine learning for additive manufacturing. Additive Manufacturing,52, 102691. https://doi.org/10.1016/j.addma.2022.102691
Ren, W., Wen, G., Zhang, Z., & Mazumder, J. (2022). Quality monitoring in additive manufacturing using emission spectroscopy and unsupervised deep learning. Materials and Manufacturing Processes,37(11), 1339–1346. https://doi.org/10.1080/10426914.2021.1906891
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual explanations from deep networks via gradient-based localization. In 2017 IEEE international conference on computer vision (ICCV). https://doi.org/10.1109/ICCV.2017.74.
Shui, C., Chen, Q., Wen, J., Zhou, F., Gagné, C., & Wang, B. (2022). A novel domain adaptation theory with Jensen–Shannon divergence. Knowledge-Based Systems,257, 109808. https://doi.org/10.1016/j.knosys.2022.109808
Škrabánek, P., & Martínková, N. (2022). Tuning of grayscale computer vision systems. Displays,74, 102286. https://doi.org/10.1016/j.displa.2022.102286
Snow, Z., Diehl, B., Reutzel, E. W., & Nassar, A. (2021). Toward in-situ flaw detection in laser powder bed fusion additive manufacturing through layerwise imagery and machine learning. Journal of Manufacturing Systems,59, 12–26. https://doi.org/10.1016/j.jmsy.2021.01.008
Tabernik, D., Šela, S., Skvarč, J., & Skočaj, D. (2020). Segmentation-based deep-learning approach for surface-defect detection. Journal of Intelligent Manufacturing,31(3), 759–776. https://doi.org/10.1007/s10845-019-01476-x
Taherkhani, K., Eischer, C., & Toyserkani, E. (2022). An unsupervised machine learning algorithm for in-situ defect-detection in laser powder-bed fusion. Journal of Manufacturing Processes,81, 476–489. https://doi.org/10.1016/j.jmapro.2022.06.074
Xie, Y., Li, B., Wang, C., Zhou, K., Wu, C. T., & Li, S. (2023). A bayesian regularization network approach to thermal distortion control in 3D printing. Computational Mechanics,72(1), 137–154. https://doi.org/10.1007/s00466-023-02270-6
Xie, Y., Li, S., Wu, C. T., Lyu, D., Wang, C., & Zeng, D. (2022). A generalized bayesian regularization network approach on characterization of geometric defects in lattice structures for topology optimization in preliminary design of 3D printing. Computational Mechanics,69(5), 1191–1212. https://doi.org/10.1007/s00466-021-02137-8
Yan, H., Ding, Y., Li, P., Wang, Q., Xu, Y., & Zuo, W. (2017). Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, https://doi.org/10.1109/CVPR.2017.107
Zhang, Y., Li, S., He, Q., Zhang, A., Li, C., Liao, Z., & Caraffini, F. (2023). An intelligent fault detection framework for FW-UAV based on hybrid deep domain adaptation networks and the hampel filter. International Journal of Intelligent Systems,2023, 1–19. https://doi.org/10.1155/2023/6608967
Acknowledgements
This work is supported by National Natural Science Foundation of China (71971181 and 72032005) and by Research Grant Council of Hong Kong (11200621). This research is supported by the International Science and Technology Cooperation Program of Guangdong Province (Project #2022A0505050047) and Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Lui, C.F., Maged, A. & Xie, M. A novel image feature based self-supervised learning model for effective quality inspection in additive manufacturing. J Intell Manuf (2023). https://doi.org/10.1007/s10845-023-02232-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10845-023-02232-y