Due to the statistical nature of process dynamics and the chaotic keyhole behavior [14], physics modeling is not able to predict completely the behavior of the process. Thus, in order to study the underlying complicated relationships between process parameters, performance indicators, and other involved variables [16, 27] affecting the formation of defects, it is important to use additional techniques. This paper proposes a 3-Stage Quality Assessment (3SQA) method based on machine learning techniques to allow for an algorithm that is able to capture the complex relations between various measurements and defects [28]. Defect detection is achieved indirectly meaning that there is not an intuitive and physics-based procedure involved, but instead their prediction is made blindly for each frame individually by involving the thermal image’s features. This is achieved through utilizing a Machine Learning (ML) model with a given quality label for each frame. Following this, another ML algorithm handles the overall decision for the quality of the seam, covering this way the temporal variations of the process.
During the first stage (Fig. 4), the most important features are extracted from the incoming image data utilizing a PCA-based algorithm combined with a Geometrical Feature Extraction (GFE) method. The output data are fed to the second stage, which consists of a classification model for the quality assessment of each frame. However, a decision on the overall quality evaluation of each stitch is required. Therefore, in the third stage, the authors have used statistical models (namely a Hidden Markov Model) aiming to deduce the overall quality of each stich.
It is important to mention that the generic framework for decision making (not limited to process level) can also be described through Dynamic Network Models [29]. Here, a decomposition of such a model, in three stages, is attempted to capture the details of process physics and to identify and predict uncertainties. These uncertainties are due to material impurities, process parameter variations, and even mechanical configuration errors [12].
The development and initial implementation of the aforementioned algorithms were carried out initially using MATLAB and real-image data obtained from [30]. Later on, these algorithms were implemented in Python using the scikit-learn library and enhanced with additional features in order to be able to handle real-time data streams. This way, it is also possible to reconfigure them and deploy them into the server system. The following subsections present the top-level structure of the feature extraction and classification methods which compose the 3SQA algorithm.
Feature extraction
Utilizing all pixels of the MWIR-camera leads to a total amount of 1024 features. The same applies to the output images of the high-resolution NIR-camera, 1,416,960 features per time step, configuring a high-dimensional vector not viable for the processing pipeline. Within this paper, two different approaches were followed for feature extraction from the incoming image data. In the first one, a self-developed image processing algorithm was created aiming to extract geometrical features of the melt pool’s temperature field. However, due to the high-dimensionality problem, another algorithm was also developed based on the PCA [31] method. The outputs of these algorithms were merged and passed to the second stage. It is worth mentioning that PCA can be retrained and extended, if needed, for very specific conditions, leading to an extra contribution to management of the uncertainty coming from the process.
Geometrical Feature Extraction (GFE) algorithm
In order to describe the geometry of the weld pool area, a threshold-based contour extraction was applied on images derived from the MWIR camera. This algorithm has been based on extraction of the first- and second-order moments of the thermal image, which was considered as a rigid body. The steps of the algorithm are listed below:
-
Read images in matrix format
-
Apply a filter, differentiating the temperature in space
-
Identify the center of the melt-pool through the moment of first order
-
Extract the moment of second order
-
Compare them with those the ideal temperature field
-
Identify position
-
Repeat for bigger defects and different locations within the field.
The standard deviation and the mean value, as moments, around the melt-pool center [16], can help indicating the existence, the size, and the position of the defect, when comparing to the case of different defect classes to that of the ideal specimen. As far as the filtering method followed for the purposes of this work is concerned, the spatial differentiation is used to enhance the differences in the variation of the captured thermal field in the presence of a defect. This step, however, is optional as it may also amplify noise in the measurements. A notch filter could then be applied [32] to isolate this variation. This procedure in total would allow the discretization of the image in smaller pieces, due to the identification of the temperature field’s variance, leading to indications of the defect’s existence and the mean filtered radiation field, which will provide information on the relevant size of the detected defects. It is worth mentioning that in order for the sensitivity of the proposed image processing techniques to be examined, different kinds of data sources were tested, implying different defects and sizes. Thus, pores/cracks were tested (Figs. 11, 12, 13 and 14) at various locations and different sizes and it was numerically proved that the method showed satisfactory results for defects formed up to 2 mm in depth, which in most of the laser welding applications, is more than adequate. In addition, the method with some alterations worked for the smallest observed pores, but the filters applied led to more noise in the signal and thus, more calculating time.
Statistical Feature Extraction (SFE) algorithm
In many problems of this kind and especially in multispectral sensory systems [33], the measured data vectors are high-dimensional, but it is generally perceived that the data lie near a lower-dimensional manifold [31]. In other words, it may be commonly accepted that high-dimensional data are multiple, indirect measurements of an underlying source, which typically cannot be directly measured. Learning a suitable low-dimensional manifold from high-dimensional data is essentially the same as learning this underlying source [31]. Therefore, in the specific approach, a PCA algorithm has been developed and implemented aiming to keep only important pixels of the images (MWIR, NIR) before feeding the classifier. The steps are cited below:
-
Read the experimental data
-
Determine the size of the datasets
-
Calculate the sample mean and standard deviations vectors
-
Standardize the data (centering and scaling of the data)
-
Derive Covariance Matrix
-
Compute the eigenvectors and eigen values
-
Transform the data
-
Derive the required components based on a cumulative variance threshold.
PCA is a useful mechanism for automated feature extractions, keeping at the same time the complexity of the algorithm at feasible levels since the idea behind it is rather simple and is based on covariance of the values of the pixels.
Quality assessment models
Support vector machines (SVM)
As it is highlighted above, the second stage of the proposed method is the development of the defect classification model and the prediction of the new part quality. After the dimensional reduction, several machine learning algorithms have been tested to classify the real experimental data. The main target has been the use of the quality labeled trials that would enable the algorithm to predict any welding defect. Thus, a supervised classification and prediction method had to be implemented. A plethora of welding trials were characterized in detail, based on the observed defects with quality labels (e.g., O.K., lack of fusion, porosity, no seam [34]). The decision on the frame’s quality was made based on crystallography methods as a part of the physical labeling process of the seam’s regions. Optical inspection was also incorporated where it was feasible while the threshold for single defect can be derived based on literature [35].
Given the fact that more than one class can be utilized [6], the decision on the frame being good or bad (“GOOD” & “NOT GOOD”) can be also made via a simple de-fuzzification rule, shown in the table below (Table 1) for two sub-classes. For instance, what is considered hereafter is penetration with confidence p1 and porosity confidence p2. Confidence has to do with the probability of the frame quality being the same with the criterion quality. In the table below, in case 1, both criteria indicate a relatively good frame quality, with confidence p values close to 1. In case 2, both criteria indicate a relatively bad frame quality, with confidence p values close to 1. In cases 3 and 4, the first and second criteria give out relatively bad frame quality, respectively. This leads to the adoption of the following fusion functions.
Table 1 De-fuzzification rules for quality decision in over one classes It can be concluded that a support vector machine with linear kernel gives relatively good results with respect to classification success rate. The support vector machine (SVM) is a supervised machine learning algorithm, which can be used for both classification and regression challenges [6]. It is used to identify a boundary of specific geometry between two or more classes.
Hidden Markov Models (HMM)
In general, Hidden Markov Models are a class of decision-making tools that assist in capturing the uncertainty in decision making. Herein, they are used to consider uncertainties as per the ways that have been aforementioned. The Maximum Likelihood criterion is utilized to assess the quality of a stitch, considering previous measurements as well as previous frames.
The conditional probability of this stitch to be by far so bad (event Bn), given the facts that:
-
Previous M frames (out of N-1) have a known status (good or bad), based on the SVM classification (event FM), 2)
-
Previous N stitches are good/bad, based on HMM classification (events P)
-
The process parameters used are X, based on the machine readings (event Xn).
Therefore, the probability that we are interested in it is given by Bayes’ theorem P(Bn | FMPXn) P(FMPXn) = P(FMPXn | Gn) P(Gn). The probability P(Gn) can be easily calculated through the total probability law, whilst both P(FMPXn) as well as P(FMPXn | Gn) can be measured easily. The definition of the HMM has been made this way so that the experiments required for the calculations of these probabilities will be easier. The overall Hidden Markov Model for both the simple and the multi-class cases can be seen in Fig. 5. With the help of a Trellis diagram, the maximum likelihood path can be identified. The Viterbi algorithm has been utilized for the estimation of the path [36] (Fig. 6).