Skip to main content

Machine Learning-based Automatic Optical Inspection System with Multimodal Optical Image Fusion Network


This paper proposes an automatic cast product surface defect detection system based on deep learning artificial intelligence technology. Application of deep learning is difficult because of the uneven surface and small defects of the cast product which are easily affected by the lighting position and angle. Therefore, three channel fusion data from an optical system that simultaneously acquires a 2D surface image and 3D shape information of the target object were obtained and used for deep learning. The mean average precision (mAP) of the proposed defect detection model using the three-channel fusion data is about 77%. And this result is greater than the 60% mAP of a defect detection model that uses single-channel data. For further optimization, we investigate a deep learning model that employs a deep learning network with multiple models, where each model trains and detects only a single type of defect. The experimental results demonstrate that the mAP of the model was improved to 88%.

This is a preview of subscription content, access via your institution.


  1. T. S. Newman and A. K. Jain, “A survey of automated visual inspection,” Computer Vision and Image Understanding, vol. 61, no. 2, pp. 231–262, March 1995.

    Article  Google Scholar 

  2. H. S. Don, K. S. Fu, C. R. Liu, and W. C. Lin, “Metal surface inspection using image processing techniques,” IEEE Transactions on Systems, Man, and Cybernetics, vol. SMC-14, no. 1, pp. 139–146, January 1984.

    Article  Google Scholar 

  3. J. Y. Kim, D. J. Yang, and C. H. Kim, “Evaluation of defects in the bonded area of shoes using an infrared thermal vision camera,” International Journal of Control, Automation, and Systems, vol. 1, no. 4, pp. 511–514, December 2003.

    MathSciNet  Google Scholar 

  4. H. Zheng, L. X. Kong, and S. Nahavandi, “Automatic inspection of metallic surface defects using genetic algorithms,” Journal of Materials Processing, vol. 125–126, no. 9, pp. 427–433, September 2002.

    Article  Google Scholar 

  5. T. Piironen, O. Silven, M. Pietikainen, T. Laitinen, and E. Strommer, “Automated visual inspection of rolled metal surfaces,” Machine Vision and Applications, vol. 3, no. 4, pp. 247–254, September 1990.

    Article  Google Scholar 

  6. Y. Okawa, “Automatic inspection of the surface defects of cast metals,” Computer Vision, Graphics, and Image Processing, vol. 25, no.1, pp. 89–112, January 1984.

    Article  Google Scholar 

  7. W. Y. Wu and C. C. Hou, “Automated metal surface inspection through machine vision,” The Imaging Science Journal, vol. 51, no. 2, pp.79–88, October 2016.

    Article  Google Scholar 

  8. C. Tikhe and J. S. Chitode, “Metal surface inspection for defect detection and classification using Gabor filter,” International Journal of Innovative Research in Science, Engineering and Technology, vol. 3, no. 3, pp. 13702–13709, June 2014.

    Google Scholar 

  9. R. Ren, T. Hung, and K. C. Tan, “A generic deep-learning-based approach for automated surface inspection,” IEEE Transactions on Cybernetics, vol. 48, no. 3, pp. 929–940, February 2017.

    Article  Google Scholar 

  10. M. Zhang, J. Wu, H. Lin, P. Yuan, and Y. Song, “The application of one-class classifier based on CNN in image defect detection,” Procedia Computer Science, vol. 114, pp. 341–348, November 2017.

    Article  Google Scholar 

  11. K. W. Ko, H. Cho, and J. H. Kim, “A neural network-based classification method for inspection of bead shape in high frequency electric resistance weld,” Transaction on Control, Automation and Systems Engineering, vol. 2, no. 3, pp. 182–188, September 2000.

    Google Scholar 

  12. W. Liu, D. Anguelov., D. Erhan, C. Szegedy, S. Reed, Y. Fu, and C. Berg, “SSD: Single shot multibox detector. In European conference on computer vision,” Proc. of European Conference on Computer Vision, pp. 21–37, October 2016.

    Google Scholar 

  13. J. Redmon and A. Farhadi, “YOLO9000: Better, faster, stronger,” Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7263–7271, December 2016.

    Google Scholar 

  14. J. Redmon and A. Farhadi, “YOLOv3: An Incremental Improvement,” arXiv preprint arXiv:1804.02767, pp. 779–788, April 2018.

    Google Scholar 

  15. S. Yanan, Z. Hui, L. Li, and Z. Hang, “Rail surface defect detection method based on YOLOv3 deep learning networks,” Proc. of Chinese Automation Congress, pp. 1563–1568, November 2018.

    Google Scholar 

  16. J. Li, Z. Su, J. Geng, and Y. Yin, “Real-time detection of steel strip surface defects based on improved YOLO detection network,” IFAC-PapersOnLine, vol. 51, no. 21, pp. 76–81, October 2018.

    Article  Google Scholar 

  17. S. H. Lee, H. S. Myong, and T. C. Chung, “Development of automatic nut inspection system using image processing,” KIPS Transactions on Computer and Communication Systems, vol. 11, no. 4, pp. 235–242, April 2004.

    Google Scholar 

  18. C. I. Moon, S. H. Choi, G. B. Kim, C. H. Kim, and W. J. Joo, “Development of a neural network classifier for the classification of surface defects of cold rolled stripsk,” Journal of the Korean Society for Precision Engineering, vol. 24, no. 4, pp. 76–83, April 2007.

    Google Scholar 

  19. H. F. Ng, “Automatic thresholding for defect detection,” Pattern Recognition Letters, vol. 27, no. 14, pp. 1644–1649, October 2006.

    Article  Google Scholar 

  20. D. M. Tsai, C. T. Lin, and J. F. Chen, “The evaluation of normalized cross correlations for defect detection,” Pattern Recognition Letters, vol. 24, no. 15, pp. 2525–2535, November 2003.

    Article  Google Scholar 

  21. Y. C. Cho, B. J. Choi, and J, O, Yoon, “A study on the development of backlight surface defect inspection system using computer vision,” Journal of the Korea Industrial Information Systems Research, vol. 12, no. 3, pp. 116–123, September 2007.

    Google Scholar 

  22. Par Kierkegaard, “Reflection properties of machined metal surfaces,” Optical Engineering, vol. 35, no. 3, pp.845–857, March 1996.

    Article  Google Scholar 

  23. T. H. Sun, C. C. Tseng, and M. S. Chen, “Electric contacts inspection using machine vision,” Image and Vision Computing, vol. 28, No.6, pp. 890–901, June 2010.

    Article  Google Scholar 

  24. D. Soukup and R. Huber-Mork, “Convolutional neural networks for steel surface defect detection from photometric stereo images,” Proc. of International Symposium on Visual Computing, vol. 8887, pp. 668–677, 2014.

    Google Scholar 

  25. R. J. Woodham, “Photometric method for determining surface orientation from multiple images,” Optical Engineering, vol. 19, no. 1, pp. 191–139, February 1980.

    Article  Google Scholar 

  26. J. I. Ser, “Illumination system for recognizing material and method of recognizing material using the same,” U.S. Patent and Trademark Office No. 10,451,547, 2019.

    Google Scholar 

  27. Optical system.

  28. B. H. Kim, D. Khan, C. Bohak, W. Choi, H. J. Lee, and M. Y. Kim, “V-RBNN based small drone detection in augmented datasets for 3D LADAR system,” Sensors, vol. 18, no. 11, pp. 3825. 2018.

    Article  Google Scholar 



Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Min Young Kim.

Additional information

Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This study has been conducted with the BK21 FOUR project funded by the Ministry of Education, Korea (4199990113966) and Basic Science Research program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2021R1A6A1A03043144). This work was partially supported by Korea Institute for Advancement of Technology (KIAT) grant funded by the Korea Government (MOTIE)(N0002428, The Competency Development Program for Industry Specialist) and by the Korea Institute of Industrial Technology as “Research equipment coordination project (kitech JJ-16-0001)”.

Jong Hyuk Lee received his B.S. degree from the Catholic University of Daegu, Daegu, Korea in 2017, and an M.S. degree from the Kyungpook National University, Daegu, Korea. He is currently pursuing a Ph.D. degree at the Kyungpook National University, His research interests include data engineering, deep learning, and inspection systems.

Byeong Hak Kim received his Ph.D. from the School of Electronic Engineering at Kyungpook National University, Daegu, Korea. He was a Senior Engineer at the SAMSUNG THALES and HANWHA Systems, Korea. He is currently a Senior Researcher at the Korea Institute of Industrial Technology. His research interests include IR and 3D imaging systems, visual object tracking, ML/DL object detection, 3D laser radar, and counter drone systems.

Min Young Kim received his B.S., M.S., and Ph.D. degrees from the Korea Advanced Institute of Science and Technology, Korea in 1996, 1998, and 2004, respectively. He worked as a Senior Researcher for Mirae Corp. from 2004 to 2005 and as a Chief Research Engineer for Kohyoung Corp. from 2005 to 2009 in the field of artificial vision systems for intelligent machines and robots. In 2009, he joined the School fo Electrical Engineering and Computer Science of the Kyungpook National University as an Assistant Professor. He is currently a Professor in the School of Electronics Engineering at the same university and is the Deputy Director of the KNU-LG Convergence Research Center and Director of the Research Center for Neurosurgical Robotic Systems. He was a visiting Associate Professor in the Department of Electrical and Computer Engineering and School of Medicine at Johns Hopkins University from 2014 to 2015. His research interest interests include visual intelligence for robotic perception and recognition of autonomous unmanned ground and aerial vehicles.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lee, J.H., Kim, B.H. & Kim, M.Y. Machine Learning-based Automatic Optical Inspection System with Multimodal Optical Image Fusion Network. Int. J. Control Autom. Syst. 19, 3503–3510 (2021).

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI:


  • Deep learning
  • defect detection
  • machine vision
  • optical system