Abstract
The final goal of this study is an integrated approach of walk-through from the classification of defect products to the feature estimation for preventing the cause of the defects in real-time or proactively. Particularly, this paper introduced the multimodal data analyses to classify the defect images of final products more accurate with inspection logs and factory process data in integrated manner.
Particularly, this paper focused on the industrial printing case in which the tiny and faint defects are difficult to be classified accurately. Motivation of this study is to clarify the possibility of image-wise classification instead of pixel-wise semantic segmentation of high annotation cost as well as low accuracy in partial of the similar shape classes in our previous study. It was introduced and numerically evaluated that various data augmentation as pre-process for imbalanced and small samples issue, image-multimodal data analyses with inspection log and/or the factory data, and ensemble of the multimodal and non-multimodal networks. As the result, the maximum accuracy of multimodal analyses is 79.37% of the model with test log and process data with 10 times augmented data. In addition, a confidence-based ensemble model with conditional branch results more accurate, 81.22% in summary of all classes. It is better than segmentation approach of pixel-wise in the previous study. Moreover, importance of additional variables is visualized as cause after multimodal analysis as aimed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Tsuji, T., Arima, S.: Automatic multi-class classification of tiny and faint printing defects based on semantic segmentation. In: Zimmermann, A., Howlett, R.J., Jain, L.C. (eds.) Human Centred Intelligent Systems. SIST, vol. 189, pp. 101–113. Springer, Singapore (2021). https://doi.org/10.1007/978-981-15-5784-2_9
Wu, M.-J., Jang, J.-S.R., Chen, J.-L.: Wafer map failure pattern recognition and similarity ranking for large-scale data sets. IEEE Trans. Semicond. Manuf. 28(1), 1–12 (2015). https://doi.org/10.1109/TSM.2014.2364237
Nakazawa, T., Kulkarni, D.V.: Wafer map defect pattern classification and image retrieval using convolutional neural network. IEEE Trans. Semicond. Manuf. 31(2), 309–314 (2018)
Kyeong, K., Kim, H.: Classification of mixed-type defect patterns in wafer bin maps using convolutional neural networks. IEEE Trans. Semicond. Manuf. 31(3), 395–401 (2018)
Jin, C.H., Na, H.J., Piao, M., Pok, G., Ryu, K.H.: A novel DBSCAN-based defect pattern detection and classification framework for wafer bin map. IEEE Trans. Semicond. Manuf. 32(3), 286–292 (2019)
Wang, R., Chen, N.: Detection and recognition of mixed-type defect patterns in wafer bin maps via tensor voting. IEEE Trans. Semicond. Manuf. 35(3), 485–493 (2022)
Duong, C.T., Lebret, R., Aberer, K.: Multimodal Classification for Analysing Social Media (2017). https://arxiv.org/abs/1708.02099. Accessed 28 Dec 2022
Woo, L.J., Yoon, Y.C.: Fine-grained plant identification using wide and deep learning model. In: 2019 International Conference on Platform Technology and Service (PlatCon). IEEE (2019)
Nakata, K., Orihara, R.: A comprehensive big-data based monitoring system for yield enhancement in semiconductor manufacturing. IEEE Trans. Semicond. Manuf. 30(4), 339–344 (2017)
Hirono, S., Uchibe, T., Murata, T., Ito, N.: Image recognition AI to promote the automation of visual inspections. Fujitsu 69(4), 42–48 (2018)
Tamaki, T.: POODL–Image recognition cloud plat form for printing factory. https://www.slideshare.net/TeppeiTamaki/poodl-a-image-recognition-cloud-platform-for-every-printing-factory. Accessed 22 Jan 2019
Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning. J. Big Data 6(1), 1–48 (2019). https://doi.org/10.1186/s40537-019-0197-0
Johnson, J.M., Khoshgoftaar, T.M.: Survey on deep learning with class imbalance. J. Big Data 6(1), 1–54 (2019). https://doi.org/10.1186/s40537-019-0192-5
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587. IEEE, Ohio (2014)
Badrinarayanan, V., Kendall, A., Cipolla, R.: SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2481–2495 (2017)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2881–2890. IEEE (2017)
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988. IEEE (2017)
Lee, R.: Dice: measures of the amount of ecologic association between species. Ecology 26(3), 297–302 (1945)
Krizhevsky, A.: The CIFAR-10 dataset (2009). https://www.cs.toronto.edu/~kriz/cifar.html. Accessed 12 Feb 2021
Cheng, H.-T., et al.: Wide & deep learning for recommender systems. In: Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, pp. 7–10 (2016)
Wei, Y., Wang, H.: Mixed-type wafer defect recognition with multi-scale information fusion transformer. IEEE Trans. Semicond. Manuf. 35(3), 341–352 (2022)
Acknowledgements
We thank for all members of the company-university collaboration research. For data collection phase, this research is supported by Cross-ministerial Strategic Innovation Promotion Pro-gram (SIP), “Big-data and AI-enabled Cyberspace Technologies” (Funding Agency: NEDO). In addition, this research has been partly executed in response to support to KIOXIA Corporation. The authors appreciate all the supports.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Itou, H., Watanabe, K., Arima, S. (2023). Image-Multimodal Data Analysis for Defect Classification: Case Study of Industrial Printing. In: Czarnowski, I., Howlett, R., Jain, L.C. (eds) Intelligent Decision Technologies. KESIDT 2023. Smart Innovation, Systems and Technologies, vol 352. Springer, Singapore. https://doi.org/10.1007/978-981-99-2969-6_4
Download citation
DOI: https://doi.org/10.1007/978-981-99-2969-6_4
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-2968-9
Online ISBN: 978-981-99-2969-6
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)