Skip to main content

Advertisement

SpringerLink
Log in

Anomaly detection in additive manufacturing processes using supervised classification with imbalanced sensor data based on generative adversarial network

  • Published:
Journal of Intelligent Manufacturing Aims and scope Submit manuscript

Abstract

Supervised classification methods have been widely utilized for the quality assurance of the advanced manufacturing process, such as additive manufacturing (AM) for anomaly (defects) detection. However, since abnormal states (with defects) occur much less frequently than normal ones (without defects) in a manufacturing process, the number of sensor data samples collected from a normal state is usually much more than that from an abnormal state. This issue causes imbalanced training data for classification analysis, thus deteriorating the performance of detecting abnormal states in the process. It is beneficial to generate effective artificial sample data for the abnormal states to make a more balanced training set. To achieve this goal, this paper proposes a novel data augmentation method based on a generative adversarial network (GAN) using additive manufacturing process image sensor data. The novelty of our approach is that a standard GAN and classifier are jointly optimized with techniques to stabilize the learning process of standard GAN. The diverse and high-quality generated samples provide balanced training data to the classifier. The iterative optimization between GAN and classifier provides the high-performance classifier. The effectiveness of the proposed method is validated by both open-source data and real-world case studies in polymer and metal AM processes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  • Arjovsky, M., & Bottou, L. (2017). Towards principled methods for training generative adversarial networks. arXiv preprint arXiv:1701.04862 .

  • Banadaki, Y., Razaviarab, N., Fekrmandi, H., & Sharifi, S. (2020). Toward enabling a reliable quality monitoring system for additive manufacturing process using deep convolutional neural networks. arXiv preprint arXiv:2003.08749 .

  • Bastani, K., Rao, P. K., & Kong, Z. (2016). An online sparse estimation-based classification approach for real-time monitoring in advanced manufacturing processes from heterogeneous sensor data. IIE Transactions, 48(7), 579–598.

    Article  Google Scholar 

  • Batista, G. E., Prati, R. C., & Monard, M. C. (2004). A study of the behavior of several methods for balancing machine learning training data. ACM SIGKDD Explorations Newsletter, 6(1), 20–29.

    Article  Google Scholar 

  • Bhat, N. N., Dutta, S., Vashisth, T., Pal, S., Pal, S. K., & Sen, R. (2016). Tool condition monitoring by svm classification of machined surface images in turning. The International Journal of Advanced Manufacturing Technology, 83(9), 1487–1502.

    Article  Google Scholar 

  • Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16, 321–357.

    Article  Google Scholar 

  • Chawla, N. V., Japkowicz, N., & Kotcz, A. (2004). Special issue on learning from imbalanced data sets. ACM SIGKDD explorations newsletter, 6(1), 1–6.

    Article  Google Scholar 

  • Chen, T., Cheng, Y., Gan, Z., Liu, J., & Wang, Z. (2021). Data-efficient gan training beyond (just) augmentations: A lottery ticket perspective. Advances in Neural Information Processing Systems 34 .

  • Choi, H.S., Jung, D., Kim, S., & Yoon. S. (2021). Imbalanced data classification via cooperative interaction between classifier and generator. IEEE Transactions on Neural Networks and Learning Systems .

  • Davenport, H. (1951). On a principle of lipschitz. Journal of the London Mathematical Society, 1(3), 179–183.

    Article  Google Scholar 

  • Dhillon, A., & Verma, G. K. (2020). Convolutional neural network: A review of models, methodologies and applications to object detection. Progress in Artificial Intelligence, 9(2), 85–112.

    Article  Google Scholar 

  • Dimitriadis, G., Neto, J. P., & Kampff, A. R. (2018). t-sne visualization of large-scale neural recordings. Neural computation, 30(7), 1750–1774.

    Article  Google Scholar 

  • Douzas, G., & Bacao, F. (2018). Effective data generation for imbalanced learning using conditional generative adversarial networks. Expert Systems with Applications, 91, 464–471.

    Article  Google Scholar 

  • Dowson, D., & Landau, B. (1982). The fréchet distance between multivariate normal distributions. Journal of Multivariate Analysis, 12(3), 450–455.

    Article  Google Scholar 

  • Fedus, W., Rosca, M., Lakshminarayanan, B., Dai, A.M., Mohamed, S., & Goodfellow, I. (2017). Many paths to equilibrium: Gans do not need to decrease a divergence at every step. arXiv preprint arXiv:1710.08446 .

  • Fernández, A., Garcia, S., Herrera, F., & Chawla, N. V. (2018). Smote for learning from imbalanced data: Progress and challenges, marking the 15-year anniversary. Journal of Artificial Intelligence Research, 61, 863–905.

    Article  Google Scholar 

  • Garcia, G. R., Michau, G., Ducoffe, M., Gupta, J. S., & Fink, O. (2022). Temporal signals to images: Monitoring the condition of industrial assets with deep learning image processing algorithms. Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability, 236(4), 617–627.

    Google Scholar 

  • Gobert, C., Arrieta, E., Belmontes, A., Wicker, R.B., Medina, F., & McWilliams, B. (2019). Conditional generative adversarial networks for in-situ layerwise additive manufacturing data. In 2019 International Solid Freeform Fabrication Symposium. University of Texas at Austin.

  • Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. Advances in Neural Information Processing Systems 27.

  • Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A.C. (2017). Improved training of wasserstein gans. Advances in Neural Information Processing Systems 30.

  • Guo, J., Wu, J., Sun, Z., Long, J., & Zhang, S. (2019). Fault diagnosis of delta 3d printers using transfer support vector machine with attitude signals. IEEE Access, 7, 40359–40368.

    Article  Google Scholar 

  • Hajalfadul, M., & Baumers, M. (2021). Building a quality cost model for additive manufacturing. University Of Khartoum Engineering Journal 11(1).

  • Han, H., Wang, W.Y., & Mao, B.H. (2005). Borderline-smote: a new over-sampling method in imbalanced data sets learning. In International conference on intelligent computing, pp. 878–887. Springer.

  • He, H., Bai, Y., Garcia, E.A., & Li, S. (2008). Adasyn: Adaptive synthetic sampling approach for imbalanced learning. In 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence), pp. 1322–1328. IEEE.

  • Hou, W., Guo, H., Yan, B., Xu, Z., Yuan, C., & Mao, Y. (2022). Tool wear state recognition under imbalanced data based on wgan-gp and lightweight neural network shufflenet. Journal of Mechanical Science and Technology, 36(10), 4993–5009.

    Article  Google Scholar 

  • Huang, G., & Jafari, A.H. (2021). Enhanced balancing gan: Minority-class image generation. Neural Computing and Applications: 1–10 .

  • Jia, Z., Liu, Z., Vong, C. M., & Pecht, M. (2019). A rotating machinery fault diagnosis method based on feature learning of thermal images. Ieee Access, 7, 12348–12359.

    Article  Google Scholar 

  • Jin, Z., Zhang, Z., & Gu, G. X. (2019). Autonomous in-situ correction of fused deposition modeling printers using computer vision and deep learning. Manufacturing Letters, 22, 11–15.

    Article  Google Scholar 

  • Kingma, D.P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 .

  • Kirka, M. M., Lee, Y., Greeley, D. A., Okello, A., Goin, M. J., Pearce, M. T., & Dehoff, R. R. (2017). Strategy for texture management in metals additive manufacturing. Jom, 69(3), 523–531.

    Article  Google Scholar 

  • Kodali, N., Abernethy, J., Hays, J., & Kira, Z. (2017). On convergence and stability of gans. arXiv preprint arXiv:1705.07215.

  • Kwon, O., Kim, H. G., Ham, M. J., Kim, W., Kim, G. H., Cho, J. H., Kim, N. I., & Kim, K. (2020). A deep neural network for classification of melt-pool images in metal additive manufacturing. Journal of Intelligent Manufacturing, 31(2), 375–386.

    Article  Google Scholar 

  • Kyeong, K., & Kim, H. (2018). Classification of mixed-type defect patterns in wafer bin maps using convolutional neural networks. IEEE Transactions on Semiconductor Manufacturing, 31(3), 395–402.

    Article  Google Scholar 

  • Li, Y., Shi, Z., Liu, C., Tian, W., Kong, Z., & Williams, C.B. (2021). Augmented time regularized generative adversarial network (atr-gan) for data augmentation in online process anomaly detection. IEEE Transactions on Automation Science and Engineering.

  • Liu, C., Kong, Z., Babu, S., Joslin, C., & Ferguson, J. (2021). An integrated manifold learning approach for high-dimensional data feature extractions and its applications to online process monitoring of additive manufacturing. IISE Transactions, 53(11), 1215–1230.

    Google Scholar 

  • Liu, C., Law, A. C. C., Roberson, D., & Kong, Z. J. (2019). Image analysis-based closed loop quality control for additive manufacturing with fused filament fabrication. Journal of Manufacturing Systems, 51, 75–86.

    Article  Google Scholar 

  • Mahmoudi, M., Ezzat, A.A., & Elwany, A. (2019). Layerwise anomaly detection in laser powder-bed fusion metal additive manufacturing. Journal of Manufacturing Science and Engineering 141(3).

  • Makes, A., & Collaborative, A.A.M.S. (2017). Standardization roadmap for additive manufacturing. February), Public Draft.

  • Mariani, G., Scheidegger, F., Istrate, R., Bekas, C., & Malossi, C. (2018). Bagan: Data augmentation with balancing gan. arXiv preprint arXiv:1803.09655.

  • Mirza, M., & Osindero, S. (2014). Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784.

  • Misra, S., Kim, D., Kim, J., Shin, W., & Kim, C. (2022). A voting-based ensemble feature network for semiconductor wafer defect classification. Scientific Reports, 12(1), 16254.

    Article  Google Scholar 

  • Montazeri, M., & Rao, P. (2018). Sensor-based build condition monitoring in laser powder bed fusion additive manufacturing process using a spectral graph theoretic approach. Journal of Manufacturing Science and Engineering 140(9).

  • Nandwana, P., & Lee, Y. (2020). Influence of scan strategy on porosity and microstructure of ti-6al-4v fabricated by electron beam powder bed fusion. Materials Today Communications, 24, 100962.

    Article  Google Scholar 

  • Odena, A., Olah, C., & Shlens, J. (2017). Conditional image synthesis with auxiliary classifier gans. In International conference on machine learning, pp. 2642–2651.

  • Powers, D.M. (2020). Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation. arXiv preprint arXiv:2010.16061.

  • Ranasinghe, G.D., & Parlikad, A.K. (2019). Generating real-valued failure data for prognostics under the conditions of limited data availability. In 2019 IEEE International Conference on Prognostics and Health Management (ICPHM), pp. 1–8. IEEE.

  • Rao, P.K., Liu, J.P., Roberson, D., Kong, Z.J., & Williams, C. (2015). Online real-time quality monitoring in additive manufacturing processes using heterogeneous sensors. Journal of Manufacturing Science and Engineering 137(6).

  • Ross, A., Willson, V. L., Ross, A., & Willson, V. L. (2017). Paired samples t-test (pp. 17–19). Writing Results Sections and Creating Tables and Figures: Basic and Advanced Statistical Tests.

    Google Scholar 

  • Sáez, J. A., Luengo, J., Stefanowski, J., & Herrera, F. (2015). Smote-ipf: Addressing the noisy and borderline examples problem in imbalanced classification by a re-sampling method with filtering. Information Sciences, 291, 184–203.

    Article  Google Scholar 

  • Saville, A. I., Vogel, S. C., Creuziger, A., Benzing, J. T., Pilchak, A. L., Nandwana, P., Klemm-Toole, J., Clarke, K. D., Semiatin, S. L., & Clarke, A. J. (2021). Texture evolution as a function of scan strategy and build height in electron beam melted ti-6al-4v. Additive Manufacturing, 46, 102118.

    Article  Google Scholar 

  • Scime, L., Siddel, D., Baird, S., & Paquit, V. (2020). Layer-wise anomaly detection and classification for powder bed additive manufacturing processes: A machine-agnostic algorithm for real-time pixel-wise semantic segmentation. Additive Manufacturing, 36, 101453.

    Article  Google Scholar 

  • Shen, B., Xie, W., & Kong, Z. J. (2020). Clustered discriminant regression for high-dimensional data feature extraction and its applications in healthcare and additive manufacturing. IEEE Transactions on Automation Science and Engineering, 18(4), 1998–2010.

    Article  Google Scholar 

  • Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep learning. Journal of Big Data, 6(1), 1–48.

    Article  Google Scholar 

  • Srivastava, A., Valkov, L., Russell, C., Gutmann, M.U., & Sutton, C. (2017). Veegan: Reducing mode collapse in gans using implicit variational learning. Advances in Neural Information Processing Systems 30 .

  • Tao, S., & Wang, J. (2020). Alleviation of gradient exploding in gans: Fake can be real. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1191–1200.

  • Tran, N.T., Bui, T.A., & Cheung, N.M. (2018). Dist-gan: An improved gan using distance constraints. In Proceedings of the European conference on computer vision (ECCV), pp. 370–385.

  • Wang, C., Yu, Z., Zheng, H., Wang, N., & Zheng B., (2017). Cgan-plankton: Towards large-scale imbalanced class generation and fine-grained classification. In 2017 IEEE International Conference on Image Processing (ICIP), pp. 855–859. IEEE.

  • Wang, R., & Chen, N. (2019). Wafer map defect pattern recognition using rotation-invariant features. IEEE Transactions on Semiconductor Manufacturing, 32(4), 596–604.

    Article  Google Scholar 

  • Wang, R., Law, A. C., Garcia, D., Yang, S., & Kong, Z. (2021). Development of structured light 3d-scanner with high spatial resolution and its applications for additive manufacturing quality assurance. The International Journal of Advanced Manufacturing Technology, 117(3), 845–862.

    Google Scholar 

  • Wang, Y., Wei, Y., & Wang, H. (2022). A class imbalanced wafer defect classification framework based on variational autoencoder generative adversarial network. Measurement Science and Technology, 34(2), 024008.

    Article  Google Scholar 

  • Wu, M. J., Jang, J. S. R., & Chen, J. L. (2014). Wafer map failure pattern recognition and similarity ranking for large-scale data sets. IEEE Transactions on Semiconductor Manufacturing, 28(1), 1–12.

    Google Scholar 

  • Xiao, H., Rasul, K., & Vollgraf, R. (2017). Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 .

  • Yu, J., & Liu, J. (2020). Two-dimensional principal component analysis-based convolutional autoencoder for wafer map defect detection. IEEE Transactions on Industrial Electronics, 68(9), 8789–8797.

  • Zhang, J., Yi, S., Liang, G., Hongli, G., Xin, H., & Hongliang, S. (2020). A new bearing fault diagnosis method based on modified convolutional neural networks. Chinese Journal of Aeronautics, 33(2), 439–447.

  • Zhang, Y. (2018). Deep generative model for multi-class imbalanced learning. University of Rhode Island.

  • Zhou, L., Tao, H., Paszke, W., Stojanovic, V., & Yang, H. (2020). Pd-type iterative learning control for uncertain spatially interconnected systems. Mathematics, 8(9), 1528.

    Article  Google Scholar 

  • Zhu, J., Yang, G., & Lio, P. (2019). How can we make gan perform better in single medical image super-resolution? a lesion focused multi-scale approach. In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp. 1669–1673. IEEE.

Download references

Acknowledgements

The research reported in this publication was supported by the Office of Naval Research under award N00014-18-1-2794, and the Department of Defense under award N00014-19-1-2728.

Author information

Authors and Affiliations

  1. Grado Department of Industrial and Systems Engineering, Virginia Tech, Blacksburg, VA, USA

    Jihoon Chung & Zhenyu James Kong

  2. Department of Mechanical and Industrial Engineering, NJIT, Newark, NJ, USA

    Bo Shen

Authors
  1. Jihoon Chung
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bo Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhenyu James Kong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhenyu James Kong.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix A: Classification results of each label in the ablation study

The classification results of each label in Table 2 are provided in Table 10. The F-score of all labels in the proposed method achieves the best performance in the ablation studies. The proposed method significantly improves the precision of the majority class and recall of minority classes where the imbalanced training data in the baseline degrades the performances. Since all the remaining data except the training data are used as testing data, a relatively small number of T-shirt than Pullover and Dress exist in testing data. It causes the overall low results of T-shirts in Precision and F-score.

Table 11 Hyperparameters of each method
Full size table

Appendix B: t-SNE results in the Ablation Studies

Figure 15 shows the t-SNE of the feature from the intermediate layer of the classifier from the proposed method in several epochs in the ablation studies. The results represent that the generated samples from the proposed method follow the features of actual data in the classifier as the epoch increases. It validates the authentic and state-distinguishable properties of the generated samples of the proposed method.

Appendix C: Structure and Hyperparameters in the Methods

In this section, the detailed structure and hyperparameters of the proposed and benchmark methods are provided. Table 11 provides the hyperparameters of each method.

Table 12 Hyperparameters of the classifier
Full size table

The gradient penalty coefficient is determined as 10 as suggested by Kodali et al. (2017); Huang and Jafari (2021). The scheduling parameter in the Cooperative GAN, is searched within a specific range ([0.1, 0.9]) following the guidelines provided in the literature (Choi et al., 2021) and selected with the values that showed the best validation performance. A batch size (m in Algorithm 1) varies along the number of actual samples in each case study to consider the computational time. Specifically, batch sizes are 100 in “Ablation studies”, 100 in “Open-source data case study”, 60 in “Polymer additive manufacturing process data case studies”, and 50 in “Metal additive manufacturing process data case studies”. Table 12 shows the hyperparameters of the classifier in case studies. Convolutional Neural Network is used for the classifier. For a fair comparison, all the methods use the same classifier as described in Table 12.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chung, J., Shen, B. & Kong, Z.J. Anomaly detection in additive manufacturing processes using supervised classification with imbalanced sensor data based on generative adversarial network. J Intell Manuf 35, 2387–2406 (2024). https://doi.org/10.1007/s10845-023-02163-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10845-023-02163-8

Keywords

Search

Navigation