Abstract
This paper proposes an automatic cast product surface defect detection system based on deep learning artificial intelligence technology. Application of deep learning is difficult because of the uneven surface and small defects of the cast product which are easily affected by the lighting position and angle. Therefore, three channel fusion data from an optical system that simultaneously acquires a 2D surface image and 3D shape information of the target object were obtained and used for deep learning. The mean average precision (mAP) of the proposed defect detection model using the three-channel fusion data is about 77%. And this result is greater than the 60% mAP of a defect detection model that uses single-channel data. For further optimization, we investigate a deep learning model that employs a deep learning network with multiple models, where each model trains and detects only a single type of defect. The experimental results demonstrate that the mAP of the model was improved to 88%.
Similar content being viewed by others
References
T. S. Newman and A. K. Jain, “A survey of automated visual inspection,” Computer Vision and Image Understanding, vol. 61, no. 2, pp. 231–262, March 1995.
H. S. Don, K. S. Fu, C. R. Liu, and W. C. Lin, “Metal surface inspection using image processing techniques,” IEEE Transactions on Systems, Man, and Cybernetics, vol. SMC-14, no. 1, pp. 139–146, January 1984.
J. Y. Kim, D. J. Yang, and C. H. Kim, “Evaluation of defects in the bonded area of shoes using an infrared thermal vision camera,” International Journal of Control, Automation, and Systems, vol. 1, no. 4, pp. 511–514, December 2003.
H. Zheng, L. X. Kong, and S. Nahavandi, “Automatic inspection of metallic surface defects using genetic algorithms,” Journal of Materials Processing, vol. 125–126, no. 9, pp. 427–433, September 2002.
T. Piironen, O. Silven, M. Pietikainen, T. Laitinen, and E. Strommer, “Automated visual inspection of rolled metal surfaces,” Machine Vision and Applications, vol. 3, no. 4, pp. 247–254, September 1990.
Y. Okawa, “Automatic inspection of the surface defects of cast metals,” Computer Vision, Graphics, and Image Processing, vol. 25, no.1, pp. 89–112, January 1984.
W. Y. Wu and C. C. Hou, “Automated metal surface inspection through machine vision,” The Imaging Science Journal, vol. 51, no. 2, pp.79–88, October 2016.
C. Tikhe and J. S. Chitode, “Metal surface inspection for defect detection and classification using Gabor filter,” International Journal of Innovative Research in Science, Engineering and Technology, vol. 3, no. 3, pp. 13702–13709, June 2014.
R. Ren, T. Hung, and K. C. Tan, “A generic deep-learning-based approach for automated surface inspection,” IEEE Transactions on Cybernetics, vol. 48, no. 3, pp. 929–940, February 2017.
M. Zhang, J. Wu, H. Lin, P. Yuan, and Y. Song, “The application of one-class classifier based on CNN in image defect detection,” Procedia Computer Science, vol. 114, pp. 341–348, November 2017.
K. W. Ko, H. Cho, and J. H. Kim, “A neural network-based classification method for inspection of bead shape in high frequency electric resistance weld,” Transaction on Control, Automation and Systems Engineering, vol. 2, no. 3, pp. 182–188, September 2000.
W. Liu, D. Anguelov., D. Erhan, C. Szegedy, S. Reed, Y. Fu, and C. Berg, “SSD: Single shot multibox detector. In European conference on computer vision,” Proc. of European Conference on Computer Vision, pp. 21–37, October 2016.
J. Redmon and A. Farhadi, “YOLO9000: Better, faster, stronger,” Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7263–7271, December 2016.
J. Redmon and A. Farhadi, “YOLOv3: An Incremental Improvement,” arXiv preprint arXiv:1804.02767, pp. 779–788, April 2018.
S. Yanan, Z. Hui, L. Li, and Z. Hang, “Rail surface defect detection method based on YOLOv3 deep learning networks,” Proc. of Chinese Automation Congress, pp. 1563–1568, November 2018.
J. Li, Z. Su, J. Geng, and Y. Yin, “Real-time detection of steel strip surface defects based on improved YOLO detection network,” IFAC-PapersOnLine, vol. 51, no. 21, pp. 76–81, October 2018.
S. H. Lee, H. S. Myong, and T. C. Chung, “Development of automatic nut inspection system using image processing,” KIPS Transactions on Computer and Communication Systems, vol. 11, no. 4, pp. 235–242, April 2004.
C. I. Moon, S. H. Choi, G. B. Kim, C. H. Kim, and W. J. Joo, “Development of a neural network classifier for the classification of surface defects of cold rolled stripsk,” Journal of the Korean Society for Precision Engineering, vol. 24, no. 4, pp. 76–83, April 2007.
H. F. Ng, “Automatic thresholding for defect detection,” Pattern Recognition Letters, vol. 27, no. 14, pp. 1644–1649, October 2006.
D. M. Tsai, C. T. Lin, and J. F. Chen, “The evaluation of normalized cross correlations for defect detection,” Pattern Recognition Letters, vol. 24, no. 15, pp. 2525–2535, November 2003.
Y. C. Cho, B. J. Choi, and J, O, Yoon, “A study on the development of backlight surface defect inspection system using computer vision,” Journal of the Korea Industrial Information Systems Research, vol. 12, no. 3, pp. 116–123, September 2007.
Par Kierkegaard, “Reflection properties of machined metal surfaces,” Optical Engineering, vol. 35, no. 3, pp.845–857, March 1996.
T. H. Sun, C. C. Tseng, and M. S. Chen, “Electric contacts inspection using machine vision,” Image and Vision Computing, vol. 28, No.6, pp. 890–901, June 2010.
D. Soukup and R. Huber-Mork, “Convolutional neural networks for steel surface defect detection from photometric stereo images,” Proc. of International Symposium on Visual Computing, vol. 8887, pp. 668–677, 2014.
R. J. Woodham, “Photometric method for determining surface orientation from multiple images,” Optical Engineering, vol. 19, no. 1, pp. 191–139, February 1980.
J. I. Ser, “Illumination system for recognizing material and method of recognizing material using the same,” U.S. Patent and Trademark Office No. 10,451,547, 2019.
Optical system. http://www.deediim.com
B. H. Kim, D. Khan, C. Bohak, W. Choi, H. J. Lee, and M. Y. Kim, “V-RBNN based small drone detection in augmented datasets for 3D LADAR system,” Sensors, vol. 18, no. 11, pp. 3825. 2018.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This study has been conducted with the BK21 FOUR project funded by the Ministry of Education, Korea (4199990113966) and Basic Science Research program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2021R1A6A1A03043144). This work was partially supported by Korea Institute for Advancement of Technology (KIAT) grant funded by the Korea Government (MOTIE)(N0002428, The Competency Development Program for Industry Specialist) and by the Korea Institute of Industrial Technology as “Research equipment coordination project (kitech JJ-16-0001)”.
Jong Hyuk Lee received his B.S. degree from the Catholic University of Daegu, Daegu, Korea in 2017, and an M.S. degree from the Kyungpook National University, Daegu, Korea. He is currently pursuing a Ph.D. degree at the Kyungpook National University, His research interests include data engineering, deep learning, and inspection systems.
Byeong Hak Kim received his Ph.D. from the School of Electronic Engineering at Kyungpook National University, Daegu, Korea. He was a Senior Engineer at the SAMSUNG THALES and HANWHA Systems, Korea. He is currently a Senior Researcher at the Korea Institute of Industrial Technology. His research interests include IR and 3D imaging systems, visual object tracking, ML/DL object detection, 3D laser radar, and counter drone systems.
Min Young Kim received his B.S., M.S., and Ph.D. degrees from the Korea Advanced Institute of Science and Technology, Korea in 1996, 1998, and 2004, respectively. He worked as a Senior Researcher for Mirae Corp. from 2004 to 2005 and as a Chief Research Engineer for Kohyoung Corp. from 2005 to 2009 in the field of artificial vision systems for intelligent machines and robots. In 2009, he joined the School fo Electrical Engineering and Computer Science of the Kyungpook National University as an Assistant Professor. He is currently a Professor in the School of Electronics Engineering at the same university and is the Deputy Director of the KNU-LG Convergence Research Center and Director of the Research Center for Neurosurgical Robotic Systems. He was a visiting Associate Professor in the Department of Electrical and Computer Engineering and School of Medicine at Johns Hopkins University from 2014 to 2015. His research interest interests include visual intelligence for robotic perception and recognition of autonomous unmanned ground and aerial vehicles.
Rights and permissions
About this article
Cite this article
Lee, J.H., Kim, B.H. & Kim, M.Y. Machine Learning-based Automatic Optical Inspection System with Multimodal Optical Image Fusion Network. Int. J. Control Autom. Syst. 19, 3503–3510 (2021). https://doi.org/10.1007/s12555-020-0118-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12555-020-0118-1