Skip to main content
SpringerLink
Log in

Deep learning-based automatic optical inspection system empowered by online multivariate autocorrelated process control

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

Defect identification of tiny-scaled electronics components with high-speed throughput remains an issue in quality inspection technology. Convolutional neural networks (CNNs) deployed in automatic optical inspection (AOI) systems are powerful for detecting defects. However, they focus on individual samples but suffer from poor process control and lack of monitoring and providing the online status regarding the production process. Integrating CNN and statistical process control models will empower high-speed production lines to achieve proactive quality inspection. With the performance of the average run length for a certain range of the shifts, the proposed control chart has high detection performance for small mean shifts in quality. The proposed control chart is successfully applied to an electronic conductor manufacturing process. The proposed model facilitates a systematic quality inspection for tiny electronics components in a high-speed production line. The CNN-based AOI model empowered by the proposed control chart enables quality checking at the individual product level and process monitoring at the system level simultaneously. The contribution of the present study lies in the proposed process control framework integrating with the CNN-based AOI model in which a residual-based mixed multivariate cumulative sum (CUSUM) and exponentially weighted moving average (EWMA) control chart for monitoring online multivariate autocorrelated processes to efficiently detect defects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability

Not applicable.

Code availability

Not applicable.

References

  1. Ojer M, Serrano I, Saiz F, Barandiaran I, Gil I, Aguinaga D, Alejandro D (2020) Real-time automatic optical system to assist operators in the assembling of electronic components. Int J Adv Manuf Technol 107(5–6):2261–2275

    Article  Google Scholar 

  2. Prieto F, Redarce T, Lepage R, Boulanger P (2002) An automated inspection system. Int J Adv Manufact Tech, 19(12), 917-925

  3. Huang SH, Pan YC (2015) Automated visual inspection in the semiconductor industry: A survey. Comput Ind 66:1–10

    Article  Google Scholar 

  4. Hung CW, Jiang JG, Wu HHP, Mao WL (2018) An Automated Optical Inspection system for a tube inner circumference state identification. J Robotics, Networking and Artificial Life, 4(4), 308-311

  5. Mar NSS, Yarlagadda PKDV, Fookes C (2011) Design and development of automatic visual inspection system for PCB manufacturing. Robot Comput Integr Manuf 27(5):949–962

    Article  Google Scholar 

  6. Taha EM, Emary E, Moustafa K (2014) Automatic Optical Inspection for PCB Manufacturing : a Survey. Int J Sci Eng Res 5(7)

  7. Liu H, Yu Y, Sun F, Gu J (2017) Visual – Tactile Fusion for Object Recognition. IEEE Trans Autom Sci Eng 14(2):996–1008

    Article  Google Scholar 

  8. Akram MW, Li G, Jin Y, Chen X, Zhu C, Zhao X, Ahmad A (2019) CNN based automatic detection of photovoltaic cell defects in electroluminescence images. Energy, 189, 116319

  9. Dai W, Mujeeb A, Erdt M, Sourin A (2020) Soldering defect detection in automatic optical inspection. Adv Eng Inform 43(November 2019):101004

  10. Lin YL, Chiang YM, Hsu HC (2018) Capacitor Detection in PCB Using YOLO Algorithm. 2018 Int Conf Syst Sci Eng ICSSE 2018 17–20

  11. Mai X, Member S, Zhang H, Jia X, Member S, Meng MQ (2020) Faster R-CNN With Classifier Fusion for Automatic Detection of Small Fruits. IEEE Trans Autom Sci Eng 17(3):1555–1569

    Google Scholar 

  12. Li W, Tsung F, Song Z, Zhang K, Xiang D (2021) Multi-sensor based landslide monitoring via transfer learning. J Qual Tech, 1-14

  13. Bersimis S, Psarakis S, Panaretos J (2007) Control Charts : An Overview. (November 2006):517–543

  14. Lyu J, Chen M (2009) Automated visual inspection expert system for multivariate statistical process control chart. Expert Systems with Applications, 3 (3), 5113-5118

  15. Zaman B, Riaz M, Abbas N, Does RJMM (2015) Mixed Cumulative Sum-Exponentially Weighted Moving Average Control Charts: An Efficient Way of Monitoring Process Location. Qual Reliab Eng Int 31(8):1407–1421

    Article  Google Scholar 

  16. Zaman B, Abbas N, Riaz M, Lee MH (2016) Mixed CUSUM-EWMA chart for monitoring process dispersion. Int J Adv Manuf Technol 86(9–12):3025–3039

    Article  Google Scholar 

  17. Zaman B, Riaz M, Lee MH (2017) On the Performance of Control Charts for Simultaneous Monitoring of Location and Dispersion Parameters. Qual Reliab Eng Int 33(1):37–56

    Article  Google Scholar 

  18. Zaman B, Lee MH, Riaz M, Abujiya MR (2020) An improved process monitoring by mixed multivariate memory control charts: An application in wind turbine field. Comput Ind Eng 142(September 2019):106343

  19. Xue L, Qiu P (2021) A nonparametric CUSUM chart for monitoring multivariate serially correlated processes. J Qual Tech, 53(4), 396-409

  20. Xu S, An X, Qiao X, Zhu L, Li L (2013) Multi-output least-squares support vector regression machines. Pattern Recogn Lett 34(9):1078–1084

    Article  Google Scholar 

  21. Ren S, He K, Girshick R, Sun J (2017) Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149

    Article  Google Scholar 

  22. Chen FC, Jahanshahi MR (2018) NB-CNN: Deep Learning-Based Crack Detection Using Convolutional Neural Network and Naïve Bayes Data Fusion. IEEE Trans Industr Electron 65(5):4392–4400

    Article  Google Scholar 

  23. Lee DT (1978) A computerized cutomatic inspection system for complex printed thick film patterns. Technical Symposium East 3:172–177

    Google Scholar 

  24. Hara Y, Akiyama N, Karasaki K (1983) Automatic Inspection System for Printed Circuit Boards. IEEE Trans Pattern Anal Mach Intell PAMI-5(6):623–630

  25. Hong JJ, Park KJ, Kim KG (1998) Parallel processing machine vision system for bare PCB inspection. IECON Proc (Ind Electron Conf) 3:1346–1350

  26. Mandeville JR (1985) Novel Method for Analysis of Printed Circuit Images. IBM J Res Dev 29(1):73–86

    Article  Google Scholar 

  27. Sun YN, Tsai CT (1992) A new model-based approach for industrial visual inspection. Pattern Recogn 25(11):1327–1336

    Article  Google Scholar 

  28. Belbachir AN, Lera M, Fanni A, Montisci A (2005) An automatic optical inspection system for the diagnosis of printed circuits based on neural networks. Conf Rec Ind Appl Soc (IEEE Industry Applications Society) 1:680–684

    Google Scholar 

  29. Weimer D, Scholz-Reiter B, Shpitalni M (2016) Design of deep convolutional neural network architectures for automated feature extraction in industrial inspection. CIRP Ann Manuf Technol 65(1):417–420

    Article  Google Scholar 

  30. Krizhevsky A, Sutskever I, Hinton GE (2017) ImageNet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  31. Sze V, Chen YH, Yang TJ, Emer JS (2017) Efficient Processing of Deep Neural Networks: A Tutorial and Survey. Proc IEEE 105(12):2295–2329

    Article  Google Scholar 

  32. Agarwal S, Terrail JOD, Jurie F (2018) Recent advances in object detection in the age of deep convolutional neural networks. arXiv preprint arXiv:1809.03193

  33. Zhao ZQ, Zheng P, Xu ST, Wu X (2019) Object Detection with Deep Learning: A Review. IEEE Trans Neural Netw Learn Syst 30(11):3212–3232

    Article  Google Scholar 

  34. Akhtar MB (2022) The Use of a Convolutional Neural Network in Detecting Soldering Faults from a Printed Circuit Board Assembly. HighTech Innov J 3(1):1–14

    Article  MathSciNet  Google Scholar 

  35. Girshick R, Donahue J, Darrell T, Malik J, Berkeley UC, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 1:5000

    Google Scholar 

  36. Huang R, Pedoeem J, Chen C (2019) YOLO-LITE: A Real-Time Object Detection Algorithm Optimized for Non-GPU Computers. Proc - 2018 IEEE Int Conf Big Data Big Data 2018 2503–2510

  37. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Unified, real-time object detection. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 779–788

  38. Redmon J, Farhadi A (2017) YOLO9000: Better, faster, stronger. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-Janua 6517–6525

  39. Redmon J, Farhadi A (2018) YOLOv3: An incremental improvement. ArXiv

  40. Khediri IB, Weihs C, Limam M (2010) Support Vector Regression control charts for multivariate nonlinear autocorrelated processes. Chemom Intell Lab Syst 103(1):76–81

    Article  Google Scholar 

  41. Psarakis S, Papaleonida GEA (2007) SPC Procedures for Monitoring Autocorrelated. Qual Reliab Eng Int 4(4):501–540

    MathSciNet  Google Scholar 

  42. Loredo EN, Jearkpaporn D, Borror CM (2002) Model-based control chart for autoregressive and correlated data.pdf. Qual Reliab Eng Int 18:489–496

  43. Atienza OO, Tang LC, Ang BW (2002) A CUSUM scheme for autocorrelated observations. J Qual Technol 34(2):187–199

    Article  Google Scholar 

  44. Li J, Jeske DR, Zhou Y, Zhang X (2019) A wavelet-based nonparametric CUSUM control chart for autocorrelated processes with applications to network surveillance. Qual Reliab Eng Int 35:644–658

    Article  Google Scholar 

  45. Zou C, Tsung F (2010) Likelihood ratio-based distribution-free EWMA control charts. J Qual Technol 42(2):174–196

    Article  Google Scholar 

  46. Zhou Q, Zou C, Wang Z, Jiang W (2012) Likelihood-based EWMA charts for monitoring poisson count data with time-varying sample sizes. J Am Stat Assoc 107(499):1049–1062

    Article  MathSciNet  MATH  Google Scholar 

  47. Roberts SW (1959) Control Chart Tests Based on Geometric Moving Averages. Technometrics 1(3):239–250

    Article  Google Scholar 

  48. Psarakis S (2015) Adaptive Control Charts: Recent Developments and Extensions. Qual Reliab Eng Int 31(7):1265–1280

    Article  MathSciNet  Google Scholar 

  49. Park J, Jun CH (2015) A new multivariate EWMA control chart via multiple testing. J Process Control 26:51–55

    Article  Google Scholar 

  50. Kang JH, Yu J, Kim SB (2016) Adaptive nonparametric control chart for time-varying and multimodal processes. J Process Control 37:34–45

    Article  Google Scholar 

  51. Ajadi JO, Riaz M (2017) Mixed multivariate EWMA-CUSUM control charts for an improved process monitoring. Commun Stat Theory Methods 46(14):6980–6993

    Article  MathSciNet  MATH  Google Scholar 

  52. Haq A, Khoo MBC (2019) New adaptive EWMA control charts for monitoring univariate and multivariate coefficient of variation. Comput Ind Eng 131:28–40

    Article  Google Scholar 

  53. Jarrett JE, Pan X (2007) The quality control chart for monitoring multivariate autocorrelated processes. Comput Stat Data Anal 51(8):3862–3870

    Article  MathSciNet  MATH  Google Scholar 

  54. Moraes DAO, Oliveira FLP, Duczmal LH, Cruz FRB (2016) Comparing the inertial effect of MEWMA and multivariate sliding window schemes with confidence control charts. Int J Adv Manuf Technol 84(5–8):1457–1470

    Google Scholar 

  55. Chiang JY, Lio YL, Tsai TR (2017) MEWMA Control Chart and Process Capability Indices for Simple Linear Profiles with Within-profile Autocorrelation. Qual Reliab Eng Int 33:1083–1094

    Article  Google Scholar 

  56. Liang W, Pu X, Xiang D (2017) A distribution-free multivariate CUSUM control chart using dynamic control limits. J Appl Stat 44(11):2075–2093

    Article  MathSciNet  MATH  Google Scholar 

  57. Crosier RB (1988) Multivariate generalizations of cumulative sum quality-control schemes. Technometrics 30(3):291–303

    Article  MathSciNet  MATH  Google Scholar 

  58. Suykens JA, Vandewalle J (1999) Least squares support vector machine classifiers. Neural Process Lett 9(3):293–300

    Article  Google Scholar 

  59. Suykens JA, Van Gestel T, De Brabanter J, De Moor B, Vandewalle JP (2002) Least squares support vector machines. World scientific

  60. Vapnik VN (1999) An overview of statistical learning theory. IEEE Trans Neural Networks 10(5):988–999

    Article  Google Scholar 

  61. Vapnik VN (2013) The Nature of Statistical Learning Theory. Springer Science & Business Media

  62. Lau KW, Wu QH (2008) Local prediction of non-linear time series using support vector regression. Pattern Recogn 41(5):1539–1547

    Article  MATH  Google Scholar 

  63. Quan T, Liu X, Liu Q (2010) Weighted least squares support vector machine local region method for nonlinear time series prediction. Appl Soft Comput 10(2):562–566

    Article  Google Scholar 

  64. Liu Z, Wu Q, Zhang Y, Philip Chen CL (2011) Adaptive least squares support vector machines filter for hand tremor canceling in microsurgery. Int J Mach Learn Cybern 2(1):37–47

    Article  Google Scholar 

  65. Lu CJ (2014) Sales forecasting of computer products based on variable selection scheme and support vector regression. Neurocomputing 128:491–499

    Article  Google Scholar 

  66. Sánchez-Fernández M, de-Prado-Cumplido M, Arenas-García J, Pérez-Cruz F (2004) SVM multiregression for nonlinear channel estimation in multiple-input multiple-output systems. IEEE Trans Signal Process 52(8):2298–2307

  67. Liu G, Lin Z, Yu Y (2009) Multi-output regression on the output manifold. Pattern Recogn 42(11):2737–2743

    Article  MATH  Google Scholar 

  68. Han Z, Liu Y, Zhao J, Wang W (2012) Real time prediction for converter gas tank levels based on multi-output least square support vector regressor. Control Eng Pract 20(12):1400–1409

    Article  Google Scholar 

  69. Keerthi SS, Lin CJ (2003) Asymptotic behaviors of support vector machines with Gaussian kernel. Neural Comput 15(7):1667–1689

    Article  MATH  Google Scholar 

  70. Lin HT, Lin CJ (2003) A study on sigmoid kernels for svm and the training of non-PSD kernels by SMO-type methods. National Taiwan University, Technical Report, Department of Computer Science

    Google Scholar 

  71. Xu S, Ma F, Tao L (2007) Learn from the information contained in the false splice sites as well as in the true splice sites using SVM. In: Proc. ISKE’07, pp 1360–1366

  72. Hsu CW, Chang CC, Lin CJ (2010) A practical guide to support vector classification. National Taiwan University, Department of Computer Science

    Google Scholar 

  73. Padilla R, Netto SL, Silva EAB (2020) A Survey on Performance Metrics for Object-Detection Algorithms, (July)

  74. Khusna H, Mashuri M, Suhartono, Prastyo DD, Lee MH, Ahsan M (2019) Residual-based maximum MCUSUM control chart for joint monitoring the mean and variability of multivariate autocorrelated processes. Prod Manuf Res 7(1):364–394

  75. Qian N (1999) On the momentum term in gradient descent learning algorithms. Neural Netw 12(1):145–151

    Article  Google Scholar 

  76. Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. Int J Comput Vision 88(2):303–338

    Article  Google Scholar 

  77. Zhu X, Gao Z (2018) An efficient gradient-based model selection algorithm for multi-output least-squares support vector regression machines. Pattern Recogn Lett 111:16–22

    Article  Google Scholar 

  78. Issam BK, Mohamed L (2008) Support vector regression based residual MCUSUM control chart for autocorrelated process. Appl Math Comput 201(1–2):565–574

    MathSciNet  MATH  Google Scholar 

  79. Chen H, Pang Y, Hu Q, Liu K (2020) Solar cell surface defect inspection based on multispectral convolutional neural network. J Intelligent Manufact, 31(2), 453-468

  80. Osei-Aning R, Abbasi SA, Riaz M (2017) Mixed EWMA-CUSUM and mixed CUSUM-EWMA modified control charts for monitoring first order autoregressive processes. Qual Tech & Quant Mngt, 14(4), 429-453

  81. Alwan LC, Roberts HV (1988) Time-series modeling for statistical process control. J Business & Eco Stats, 6(1), 87-95

  82. Woodall WH, Faltin FW (1993) Autocorrelated data and SPC. ASQC Statistics Division Newsletter, 13(4), 18-21

  83. Chatterjee S, Qiu P (2009) Distribution-free cumulative sum control charts using bootstrap-based control limits. The Annals of Appl Stats, 3(1), 349-369

  84. Montgomery DC, Mastrangelo CM (1991) Some statistical process control methods for autocorrelated data. J Qual Tech, 23(3), 17 -193

  85. Harris TJ, Ross WH (1991) Statistical process control procedures for correlated observations. The Canadian J Chem Engr, 69(1), 48-57

  86. Issam BK, Mohamed L (2008) Support vector regression based residual MCUSUM control chart for autocorrelated process. Appl Math and Comp, 201(1-2), 565-574

  87. Prats-Montalbán JM, Ferrer A (2014) Statistical process control based on Multivariate Image Analysis: A new proposal for monitoring and defect detection. Comp & Chem Engr, 71, 501-511

  88. Hotelling H (1947) Multivariate quality control. Techniques of Statistical Analysis

  89. Woodall WH, Ncube MM (1985) Multivariate CUSUM quality-control procedures. Technomet, 27(3), 285-292

  90. Healy JD (1987) A note on multivariate CUSUM procedures. Technomet, 29(4), 409-412

  91. Lowry CA, Woodall WH, Champ CW, Rigdon SE (1992) A multivariate exponentially weighted moving average control chart. Technomet, 34(1), 46-53

  92. Dyer J, Conerly M, Adams BM (2003) A simulation study and evaluation of multivariate forecast based control charts applied to ARMA processes. J Statistical Comp and Simulation, 73(10), 709-724

  93. Kalgonda AA, Kulkarni SR (2004) Multivariate quality control chart for autocorrelated processes. J Appl Stats, 31(3), 317-327

  94. Efron B, Tibshirani RJ (1993) An introduction to the bootstrap. 57 Boca Raton

Download references

Funding

This work is partially supported by the Ministry of Science and Technology and Ministry of Education, R.O.C. (Taiwan) Grant ID: MOST 107—2221—E—011—101—MY3.

Author information

Authors and Affiliations

  1. Department of Industrial Management, Artificial Intelligence for Operations Management Research Center, National Taiwan University of Science and Technology, Taipei, 106, Taiwan

    Kung-Jeng Wang & Luh Juni Asrini

  2. Department of Industrial Engineering, Widya Mandala Surabaya Catholic University, Surabaya, 60114, Indonesia

    Luh Juni Asrini

Authors
  1. Kung-Jeng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luh Juni Asrini
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Kung-Jeng Wang: Conceptualization, methodology, supervision. Luh Juni Asrini: Methodology, writing-original draft preparation, programming, experiments, data analysis.

Corresponding author

Correspondence to Kung-Jeng Wang.

Ethics declarations

Ethics approval

Not applicable.

Consent to participate

This manuscript is the authors' original work and has not been published nor has it been submitted simultaneously elsewhere.

Consent for publication

All authors have checked the manuscript and have agreed to the submission.

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1. Sensitivity analysis to verify the proposed R-MMCE

Table 4 Summary of input selection and residual assumption of MLS-SVR model
Full size table
Table 5 Summary of hyper-parameter and residuals characteristics of MLS-SVR model
Full size table
Table 6 The UCL of the proposed R-MMCE control chart given \(\lambda ,\) k, and m
Full size table

Appendix 2. Summary of simulation scenario and monitoring result of the R-MMCE control chart

Dataset

Parameters

Hyper-parameter

MSE

Monitoring results

 

Training

Testing

\(\gamma^{'}\)  

\(\gamma^{"}\)  

\(\sigma\)

  

1

\({\mathbf\mu}_{a}={\mathbf 0}_4\) ; \({\mathbf\Sigma}_a={\boldsymbol I}_4\)  

\({\mathbf\mu}_{e}={\mathbf 0}_4\) ; \({\mathbf\Sigma}_e={\boldsymbol I}_4\)  

\({2}^{-5}\)

\({2}^{-6}\)

\({2}^{-1}\)

0.005

In-control

2

\({\mathbf\mu}_{a}={\mathbf 0}_4\) ; \({\mathbf\Sigma}_a={\boldsymbol I}_4\)  

\({\mathbf\mu}_{e}={\left[\begin{array}{cc}\begin{array}{ccc}1.5& 1.5& 1.5\end{array}& 1.5\end{array}\right]}^{T}\) ; \({\mathbf\Sigma}_e={\boldsymbol I}_4\)  

\({2}^{-5}\)

\({2}^{-10}\)

\({2}^{-3}\)

0.006

m+ start from sample 169-th

3

\({\mathbf\mu}_{a}={\mathbf 0}_4\) ; \({\mathbf\Sigma}_a={\boldsymbol I}_4\)  

\({\mathbf\mu}_{e}={\left[\begin{array}{cc}\begin{array}{ccc}1.5& 1.5& 1.5\end{array}& 1.5\end{array}\right]}^{T};\)  

\({\mathbf\Sigma}_e=\left[\begin{array}{cc}\begin{array}{c}\begin{array}{ccc}2.5& 1.5& 1.5\\ 1.5& 2.5& 1.5\\ 1.5& 1.5& 2.5\end{array}\\ \begin{array}{ccc}1.5& 1.5& 1.5\end{array}\end{array}& \begin{array}{c}\begin{array}{c}1.5\\ 1.5\\ 1.5\end{array}\\ 2.5\end{array}\end{array}\right]\)  

\({2}^{-5}\)

\({2}^{-4}\)

\({2}^{1}\)

0.005

m+ start from sample 179-th

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, KJ., Asrini, L.J. Deep learning-based automatic optical inspection system empowered by online multivariate autocorrelated process control. Int J Adv Manuf Technol 120, 6143–6162 (2022). https://doi.org/10.1007/s00170-022-09161-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-022-09161-9

Keywords

Search

Navigation