1 Introduction

Additive manufacturing in general and metal additive manufacturing in particular is held as a key technology for the decentralization and digitalization of manufacturing [20]. Compared to conventional manufacturing, however, it has so far remained a niche technology—Laser Powder Bed Fusion (LPBF, also PBF-LB/M), as one of the most popular metal additive manufacturing processes, holds high potential especially for lightweight applications or biomedical implants but suffers from cost, poor surface quality and lack of reproducibility [18]. The complexity of the process along with the completely different nature of crystal structure formation compared to traditional forging or casting techniques still poses a significant challenge in terms of reproducibility and quality assurance [2, 14, 15].

For metal additive manufacturing to gain industrial relevance, product qualification has to be ensured already during the process without the need for long and expensive post-process procedures (e.g. X-ray computed tomography) [5]. The process monitoring techniques most relevant for industrial use will be those that allow in situ determination of part properties. A range of process monitoring modalities aiming to allow more reliable manufacturing processes have been a major research focus in the past years [10].

Electromagnetic as well as acoustic process emissions have been recorded with a variety of sensor technologies and were linked to process quality with diverging success. Depending on whether the sensor is integrated on- or off-axially, it receives emissions from either only the melt pool or the entire build platform. Spatially integrated on-axis sensors like photodiodes or pyrometers are the most widely adapted monitoring tools on the market [16, 23]. Reference signals are gathered for certain parts and materials and compared to the signals measured in the process at hand. Besides, spatially resolved sensors like high-speed or bolometric cameras have been applied as well, both on- and off-axially [4, 12]. The process emissions could be linked to material properties and pore formation.

There are yet other approaches providing spectral information about the process zone [24], but sacrificing temporal and spatial information. In optical coherence tomography (OCT), low-coherence interferometry allows to detect surface topologies during the process, which has been used to detect subsurface defects [6]. Fringe detection, on the other hand, has been shown to be suitable for detecting coating errors [27]. In some studies, the data acquired has been evaluated using simple methods of machine learning like a support vector machine [9, 21] or advanced methods of deep learning like complex convolutional neural networks [17, 26].

None of the methods described, however, could successfully be implemented for on-line product qualification within an industrial environment [10]. This motivates the new approach presented below using hyperspectral imaging to predict part properties using a convolutional neural network.

Laser powder bed fusion of magnesium and magnesium alloys is inherently difficult. Due to the high oxidation tendency of magnesium, MgO, which has a relatively high melting point (2800 ∘C) compared to the evaporation point of pure magnesium (1090 ∘C) [11], has to be molten in every layer of the process. The process window is thus very small and smoke formation is very common. Some researchers reported using higher pressures and atmospheres enriched with hydrogen to tackle these challenges [7]. Despite all, especially the magnesium alloy WE43 with its good bioresorbability and compatibility is considered a material with high potential for additive manufacturing of biomedical implants [1]. This is why the investigations about a novel hyperspectral monitoring presented in this article were conducted while processing this alloy.

The main goal of this work is to investigate the potential of hyperspectral imaging for on-line product qualification in laser powder bed fusion. Using hyperspectral data with a convolutional neural network to predict industrially relevant part properties is a quality assurance approach that has not yet been reported elsewhere. The prediction of the roughness Rz as presented herein is only the first step towards predicting a range of spatially resolved part properties. Section 2 of this article explains the sensor technology (2.1) and provides powder (2.2) and machine (2.3) specifics. Subsections 2.4 and 2.5 give insight into how the data was acquired and processed as well as into how the surface roughness was determined, respectively. Section 3 presents results of the data evaluation by convolutional neural networks and a discussion thereof. Section 4 finally draws a conclusion including an assessment of the findings and an outlook on future research.

2 Materials and methods

2.1 Area-scan hyperspectral camera

The area-scan hyperspectral camera MQO22HG-IM-SM5X5-NIR from Ximea (Table 1) was used in this work to capture process emissions. The special characteristic that makes this sensor unique is a coating of the CMOS sensor with 25 different Fabry-Pérot interference filters that each covers an individual pixel. These filters are arranged in 5 * 5 mosaics covering the 2048 * 1088 pixels from the entire sensor resulting in a reduced spatial resolution of 409 * 217 but with added spectral resolution represented by 25 channels. With a 875-nm short pass filter protecting the pixels from laser light reflections at 1070 nm, these 25 spectral bands lie between 600 and 875 nm. The filter also annihilates the second harmonic in the Fabry-Pérot filter’s spectral transmittance.

Table 1 Specifications of hyperspectral sensor
Full size table

Figure 1a illustrates the pattern of the Fabry-Pérot interference filters on the CMOS sensor chip. The height of the filters correlates with the wavelength they are transmissive to. Figure 1b shows the plot of the quantum efficiencies of the 25 bands taking into account the effect of the short pass filter. The sensitivity in one pixel is not confined to one narrow wavelength band around the peak, but has several minor peaks at other wavelengths. This effect can be accounted for by a so-called correction matrix. Besides, the configuration is prone to noise from crosstalk induced by non-parallel light beams. Further noise is caused by dark current, vignetting effects as well as chromatic aberration of the optical setup.

Fig. 1
figure 1

Illustration of 5 * 5 mosaic patterns and spectral characteristics of hyperspectral camera

Full size image

The major benefit of this sensor technology, namely the combination of spectral as well as spatial resolution, comes with certain costs: comprehensive correction/calibration, reduced spatial and temporal resolution when compared to high-speed cameras [25], and reduced spectral resolution when compared to spectrometers [13]. Area-scan hyperspectral imaging thus fulfills a niche whose potential shall be evaluated and discussed in the remainder of this article.

2.2 Powder properties

Since powder properties strongly affect the reproducibility and quality of the LPBF process [19, 22], several powder characteristics were determined. Table 2 summarizes the crucial properties. The LPBF process is most reliable as long as powder particles are spherical and hence show good flowability. The SEM images in Fig. 2 give a qualitative reference for the particle’s shape whereas the so-called Dynamic Particle Image Analysis provides a quantitative measure for the particles sphericity. These results along with the Hausner-ratio imply an adequate flowability even though the SEM image reveals some fine particles that usually impede flowability [3]. It is also critical that the powder be sufficiently dry which was shown in the Karl Fischer Titration method.

Table 2 Magnesium alloy WE43: crucial properties of the powder material
Full size table
Fig. 2
figure 2

SEM images from WE43 powder particles

Full size image

2.3 Machine setup

The experiments were conducted with a laboratory LPBF machine providing high flexibility with regard to process control as well as hardware accessibility. The machine is schematically shown in Fig. 3a where the camera is integrated off-axially. The features of the machine are summarized in Table 3.

Table 3 Machine setup for of the magnesium alloy WE43
Full size table

The area-scan hyperspectral camera was integrated off-axially at a tilt angle of 20∘ such that the depth of focus of was sufficient to ensure a sharp image. The objective’s working distance was fixed at 190 mm, the magnification ×1.0, and its field of view (FOV) thus corresponding to the sensor size. Figure 3b shows the orientation of the manufactured part on the magnesium AZ31 substrate. One rectangular layer was built with the dimensions 14 mm * 10 mm with the camera’s FOV in the center with dimensions of 10 mm * 6 mm. This way, a maximum number of frames is collected from the center of the scan vectors. During the LPBF process, all parameters except laser power and scan speed were kept constant.

Fig. 3
figure 3

a Sketch of experimental setup for LPBF process monitoring; b magnesium substrate with laser melted part and field of view of the camera

Full size image

2.4 Data acquisition and preprocessing

Data acquisition and preprocessing scripts were implemented using the camera manufacturer’s application programming interface for the programming language Python. This was beneficial for subsequent data evaluation through methods of machine learning since these also rely on Python. Figure 4 shows schematically how data acquisition, storage and subsequent evaluation were organized. At the start of the process, the frames were continually captured and the maximum grey value was determined. As soon as the laser spot reached the FOV of the camera, this value quickly shot up. The storage of the images therefore started when the maximum grey value had reached a threshold of 20 for at least 20 ms (at an exposure time of 4 ms this corresponds to five consecutive frames). The frames were then saved at full resolution as a numpy-array data type and were successively put into a list until the grey value dropped below 20 for 20 ms. This was the case when the laser spot moved out of the camera’s FOV and the coating process started. Then, the first data processing occurred: for every frame, the pixel coordinates of the maximum grey value were determined which then served as the center for converting the 2048 * 1088 frame to a 100 * 100 frame. The cropping was then done such that the spectral bands’ pixel locations were identical for each post-processed frame. Efficient data processing requires manageable amounts of data and so the rest of the image was discarded.

Fig. 4
figure 4

Illustration of data acquisition and processing during and after the LPBF process

Full size image

All frames that were captured in one layer were saved after cropping in a so-called pickle-file that served as a container of the Python list. The processing of the frames is usually interrupted as soon as the next layer is built. For the experiments presented here, however, only one layer was built. The following data processing happened only after the build job has finished. At first, the 100 * 100 * 1 frames were dissected into 20 * 20 * 25 frames to sort the different spectral channels. As soon as the samples were investigated and the quantities of interest were determined, each frame could be assigned a label—in this case the surface roughness Rz. The result was a Python list where each element was again a list containing the frame as a numpy-array and the label as a float. This list could be used as input for a convolutional neural network.

2.5 Measurement of surface roughness

The target value of the investigations was the surface roughness of additively manufactured WE43 alloy. In order to keep ambient conditions as reproducible as possible, only one layer was manufactured while the process was monitored with the hyperspectral camera. Three parameter combinations of laser power and scan speed were selected from literature [8]. One was reported to lead to dense parts and good reproducibility (50 W; 75 mm/s), one to too much energy input (50 W; 40 mm/s), and the final one to too little energy input (20 W; 75 mm/s). The roughness Rz was measured according to DIN EN ISO 4288 using a Keyence VK X1050 confocal microscope. The cut-off wavelengths λC and λS were set to 2.5 mm and 8 μm, respectively.

The measuring section ln was therefore 12.5 mm long. With a FOV of 10 mm * 6 mm, the roughness was measured only along one direction so that most of the measuring section lay within the FOV. Figure 5a shows the confocal image of the first sample and the location of the five measuring sections. The average of the five roughness values was taken as the final value that was used as a label for all the camera images taken at the respective sample. For each sample, the model is thus provided with a large number of images that are labeled with the identical roughness value. This means that the model is trained to find features in the hyperspectral data hinting at global process characteristics leading to a certain global roughness.

Fig. 5
figure 5

The hyperspectral images are dissected into 25 channels to be used as input for the convolutional neural network. The images are labeled with the surface roughness Rz

Full size image

3 Results and discussion

3.1 Surface roughness

Table 4 shows the surface roughness measured of the different samples. Sorted by parameter set, for each sample, the average measured roughness Rz is given along with the number of frames acquired from the hyperspectral camera for this particular sample in brackets. The right column finally displays the mean roughness for the parameter set and the mean absolute error (or deviation) of that set. These values will help evaluate the prediction power of the convolutional neural network used subsequently.

Table 4 Overview over measured surface roughnesses of all samples fabricated with three different parameter sets
Full size table

3.2 Predicting the surface roughness using a convolutional neural network

The correlation between the surface roughness and the hyperspectral process recordings shall be determined. As laid out in previous sections, the images contain spatial as well as spectral information about the process emissions. This information could potentially contain crucial features for the prediction of the surface roughness. Since these features are unknown, a method from machine learning, a so-called convolutional neural network, is applied. The hyperspectral measurement is done in the high-temperature melt pool environment. Therefore, the temperature has an obvious influence on the hyperspectral measurement. Since the training data is labeled with the roughness, however, the model is trained to extract precisely the features related to roughness, not temperature.

Typically, the input data for a CNN are images with one or three color channels. The process recordings in this case, however, entail 25 channels, such that the network topology is adapted accordingly. Therefore, pre-trained networks that have achieved exceedingly high classification accuracies in other areas (e.g. ResNet-models) cannot be applied in this case.

On the one hand, the CNN shall provide a high prediction accuracy while on the other hand it may not require more computation power than was available, both for the training and for the later classification process. That is why an approach with four convolutional layers was chosen (Fig. 6) whose number of channels is doubled in each except for the last layer. The dimension of the convolutional matrix was fixed at 3 * 3. The ReLU-function served as activation function. The input data in the form of the process images entailed a large number of channels (25) but was otherwise very small (20 * 20) which is why no max-pooling layers were applied. To avoid overfitting, a dropout layer was added to the network followed by a fully connected layer before the final output is given. The mean squared error was chosen as a loss function to penalize high deviations between prediction and label during training. As evaluation metric, the mean absolute error was used, so that it could directly be compared to the values in Table 4. To compare the prediction power of different network topologies, a hyperparameter tuning was conducted during which the dropout rate (0.4 and 0.6), the number of perceptrons in the fully connected layer (8, 16, 32), and the optimizer (Adam and Nadam) were varied. Table 5 shows the results of the training for 10 epochs, respectively, with a batch size of 20 with the corresponding hyperparameters. Of the data available, 70% was used as training, 20% as validation, and the remaining 10% as test data. The test data was used to determine the prediction accuracies shown.

Fig. 6
figure 6

Network topology of the CNN used for the prediction of the surface roughness. During hyperparameter tuning, the dropout rate (0.4; 0.6) the optimizer (Adam; Nadam) as well as the number of perceptrons in the fully connected layer were varied, affecting the number of trainable parameters

Full size image
Table 5 Prediction accuracy of network topologies with a different number of perceptrons in the fully connected layer, dropout rate, and optimizer
Full size table

For better clarity, the results are visualized in Fig. 7, such that the influence of individual hyperparameters becomes apparent. It can be seen, for instance, that the Nadam-optimizer allows higher prediction accuracies than the Adam optimizer. The smallest mean absolute error and thus the highest prediction accuracy was achieved using the Nadam optimizer, a dropout rate of 0.6 and 16 perceptrons in the fully connected layer. A simplified scheme of this network topology is shown in Fig. 6.

Fig. 7
figure 7

Illustration of the hyperparameter tuning

Full size image

With these hyperparameters, a longer training process of 40 epochs was carried out. To ensure that for each parameter set the same proportion of data went into training, validation and test set, the parameter sets were divided separately. This means that for each parameter set, 70% of the data was allocated to the training, 20% to the validation, and 10% to the test data set. Only after that, the training and validation data was shuffled. The test data was kept separate so that the prediction accuracy could be evaluated for each parameter set individually. This allows to determine whether the CNN really recognizes different surface roughnesses or only distinguishes between parameter sets.

Figure 8 shows the evolution of the loss function during the training process for training and validation data. Despite the high dropout rate of 0.6, overfitting can be observed starting at around epoch 22. The network learns the features of the training data “too well”.

Fig. 8
figure 8

Evolution of the loss function (mean squared error) for training and validation set during training of the CNN

Full size image

Table 6 shows how accurately the CNN can predict the surface roughness based on the test data. For the complete data set, all three parameter sets were used. The prediction accuracy of the CNN can be expressed by an MAE of 4.1 μm which is much lower than the MAE of the roughness within the data set of 14.3 μm. It should be noted that the results of parameter set A contribute slightly more to this result than the other parameter sets due to sheer number of images in each set. As a next step, the separate test data containing only frames from one parameter set were used as input for the CNN. Table 6 shows that for parameter set A (50 W; 75 mm/s), the parameter itself is recognized, but the roughness cannot be predicted more accurately than is already implied by the statistical spread within that set. For parameter set B (50 W; 40 mm/s), on the other hand, the roughness can be predicted more accurately than implied by the inherent spread. For this set, there must be features in the hyperspectral data that contain information about the surface roughness. Parameter set C (20 W; 75 mm/s) shows a high spread and the roughness can be predicted even less accurately.

Table 6 Prediction accuracy of the convolutional neural network for different parameter sets compared to the inherent statistical spread within each set
Full size table

First of all, these results show that the different parameters can be recognized fairly well. Within the different parameter sets, however, the surface roughness Rz can be predicted with diverging accuracy when compared to the statistical spread in that set. The spread of the roughness in parameter set A is very small, making a more accurate prediction difficult. Parameter set C consists of a very low laser power, so that too little light was available for a more accurate prediction. Parameter set B, on the other hand, contains sufficient statistical spread as well as enough light so that the prediction is more accurate than the statistical spread. Besides, the largest amount of training data was available for this parameter set.

4 Conclusion

The aim of the research presented in this article was to investigate the correlation of hyperspectral recordings from the process and part properties, in this case the surface roughness Rz. To this end, the area-scan hyperspectral camera was adapted to the process and integrated into a laboratory machine. As a material particularly relevant for biomedical applications, the magnesium alloy WE43 was used in the experiments. The most relevant powder properties like particle size distribution, sphericity and moisture content were determined to ensure a reliable process.

The area-scan hyperspectral camera is a compromise between spatial and spectral resolution and as such generates large amounts of data. The data processing methodology developed in this work has proven to enable efficient data handling nonetheless by extracting only the relevant parts. In order to establish the link between the multi-channel data and the surface roughness, a convolutional neural network was adapted to the input and a hyperparameter tuning was carried out. During training of the final network topology for 40 epochs, overfitting was observed despite an integrated dropout layer. The overall prediction accuracy over the entire data including three different parameter sets corresponds to an MAE of 4.1 μm compared to an MAE of the actual roughness of 14.3 μm. The MAE of the prediction within a parameter set, however, is not always smaller than that of the actual roughness in that set.

Overall, the results suggest that hyperspectral data does provide crucial information about the LPBF process and that hyperspectral imaging could potentially help establish on-line product qualification. It has yet to be investigated, however, whether other part properties apart from the surface roughness and spatially resolved process defects can be predicted as well. This will also show whether the compromises in terms of spatial, spectral and temporal resolution that go along with this kind of sensor are justified. This should be the subject of future research in this area.