Hyperspectral imaging for prediction of surface roughness in laser powder bed fusion

This article discusses the relevance of in situ quality assurance in metal additive manufacturing for cost-efficient product qualification. It presents an approach for monitoring the laser powder bed fusion (LPBF) process using an area-scan hyperspectral camera to predict the surface roughness Rz with the help of a convolutional neural network. These investigations were carried out during LPBF processing of the magnesium alloy WE43 that, due to its bioresorbability and compatibility, holds significant potential for biomedical implants. A data acquisition and processing methodology has been set up to enable efficient management of the hyperspectral data. The hyperspectral images obtained from the process were labeled with the surface roughness Rz as determined by a confocal microscope. The data was used to train a convolutional neural network whose hyperparameters were optimized in a hyperparameter tuning process. The resulting network was able to predict the surface roughness within a mean absolute error (MAE) of 4.1 μm over samples from three different parameter sets. Since this is significantly smaller than the spread of the actual roughness measured (MAE = 14.3 μm), it indicates that the network identified features in the hyperspectral data linking to the roughness. These results provide the basis for future research aiming to link hyperspectral process images to further part properties relevant for quality assurance.


Introduction
Additive manufacturing in general and metal additive manufacturing in particular is held as a key technology for the decentralization and digitalization of manufacturing [20]. Compared to conventional manufacturing, however, it has so far remained a niche technology-Laser Powder Bed Fusion (LPBF, also PBF-LB/M), as one of the most popular metal additive manufacturing processes, holds high potential especially for lightweight applications or biomedical implants but suffers from cost, poor surface quality and lack of reproducibility [18]. The complexity of the process along with the completely different nature of crystal structure formation compared to traditional forging or casting Niklas Gerdes n.gerdes@lzh.de 1 Laser Zentrum Hannover e.V., Hollerithallee 8,30419 Hannover, Germany techniques still poses a significant challenge in terms of reproducibility and quality assurance [2,14,15].
For metal additive manufacturing to gain industrial relevance, product qualification has to be ensured already during the process without the need for long and expensive post-process procedures (e.g. X-ray computed tomography) [5]. The process monitoring techniques most relevant for industrial use will be those that allow in situ determination of part properties. A range of process monitoring modalities aiming to allow more reliable manufacturing processes have been a major research focus in the past years [10].
Electromagnetic as well as acoustic process emissions have been recorded with a variety of sensor technologies and were linked to process quality with diverging success. Depending on whether the sensor is integrated on-or offaxially, it receives emissions from either only the melt pool or the entire build platform. Spatially integrated on-axis sensors like photodiodes or pyrometers are the most widely adapted monitoring tools on the market [16,23]. Reference signals are gathered for certain parts and materials and compared to the signals measured in the process at hand. Besides, spatially resolved sensors like high-speed or bolometric cameras have been applied as well, both on-and off-axially [4,12]. The process emissions could be linked to material properties and pore formation.
There are yet other approaches providing spectral information about the process zone [24], but sacrificing temporal and spatial information. In optical coherence tomography (OCT), low-coherence interferometry allows to detect surface topologies during the process, which has been used to detect subsurface defects [6]. Fringe detection, on the other hand, has been shown to be suitable for detecting coating errors [27]. In some studies, the data acquired has been evaluated using simple methods of machine learning like a support vector machine [9,21] or advanced methods of deep learning like complex convolutional neural networks [17,26].
None of the methods described, however, could successfully be implemented for on-line product qualification within an industrial environment [10]. This motivates the new approach presented below using hyperspectral imaging to predict part properties using a convolutional neural network.
Laser powder bed fusion of magnesium and magnesium alloys is inherently difficult. Due to the high oxidation tendency of magnesium, MgO, which has a relatively high melting point (2800 • C) compared to the evaporation point of pure magnesium (1090 • C) [11], has to be molten in every layer of the process. The process window is thus very small and smoke formation is very common. Some researchers reported using higher pressures and atmospheres enriched with hydrogen to tackle these challenges [7]. Despite all, especially the magnesium alloy WE43 with its good bioresorbability and compatibility is considered a material with high potential for additive manufacturing of biomedical implants [1]. This is why the investigations about a novel hyperspectral monitoring presented in this article were conducted while processing this alloy.
The main goal of this work is to investigate the potential of hyperspectral imaging for on-line product qualification in laser powder bed fusion. Using hyperspectral data with a convolutional neural network to predict industrially relevant part properties is a quality assurance approach that has not yet been reported elsewhere. The prediction of the roughness R z as presented herein is only the first step towards predicting a range of spatially resolved part properties. Section 2 of this article explains the sensor technology (2.1) and provides powder (2.2) and machine (2.3) specifics. Subsections 2.4 and 2.5 give insight into how the data was acquired and processed as well as into how the surface roughness was determined, respectively. Section 3 presents results of the data evaluation by convolutional neural networks and a discussion thereof. Section 4 finally draws a conclusion including an assessment of the findings and an outlook on future research.

Area-scan hyperspectral camera
The area-scan hyperspectral camera MQO22HG-IM-SM5X5-NIR from Ximea (Table 1) was used in this work to capture process emissions. The special characteristic that makes this sensor unique is a coating of the CMOS sensor with 25 different Fabry-Pérot interference filters that each covers an individual pixel. These filters are arranged in 5 * 5 mosaics covering the 2048 * 1088 pixels from the entire sensor resulting in a reduced spatial resolution of 409 * 217 but with added spectral resolution represented by 25 channels. With a 875-nm short pass filter protecting the pixels from laser light reflections at 1070 nm, these 25 spectral bands lie between 600 and 875 nm. The filter also annihilates the second harmonic in the Fabry-Pérot filter's spectral transmittance. Figure 1a illustrates the pattern of the Fabry-Pérot interference filters on the CMOS sensor chip. The height of the filters correlates with the wavelength they are transmissive to. Figure 1b shows the plot of the quantum efficiencies of the 25 bands taking into account the effect of the short pass filter. The sensitivity in one pixel is not confined to one narrow wavelength band around the peak, but has several minor peaks at other wavelengths. This effect can be accounted for by a so-called correction matrix. Besides, the configuration is prone to noise from crosstalk induced by non-parallel light beams. Further noise is caused by dark current, vignetting effects as well as chromatic aberration of the optical setup.
The major benefit of this sensor technology, namely the combination of spectral as well as spatial resolution, comes with certain costs: comprehensive correction/calibration, reduced spatial and temporal resolution when compared to  high-speed cameras [25], and reduced spectral resolution when compared to spectrometers [13]. Area-scan hyperspectral imaging thus fulfills a niche whose potential shall be evaluated and discussed in the remainder of this article.

Powder properties
Since powder properties strongly affect the reproducibility and quality of the LPBF process [19,22], several powder characteristics were determined. Table 2 summarizes the crucial properties. The LPBF process is most reliable as long as powder particles are spherical and hence show good flowability. The SEM images in Fig. 2 give a qualitative reference for the particle's shape whereas the so-called Dynamic Particle Image Analysis provides a quantitative measure for the particles sphericity. These results along with the Hausner-ratio imply an adequate flowability even though the SEM image reveals some fine particles that usually impede flowability [3]. It is also critical that the powder be sufficiently dry which was shown in the Karl Fischer Titration method.

Machine setup
The experiments were conducted with a laboratory LPBF machine providing high flexibility with regard to process control as well as hardware accessibility. The machine is schematically shown in Fig. 3a where the camera is integrated off-axially. The features of the machine are summarized in Table 3.
The area-scan hyperspectral camera was integrated offaxially at a tilt angle of 20 • such that the depth of focus of was sufficient to ensure a sharp image. The objective's working distance was fixed at 190 mm, the magnification ×1.0, and its field of view (FOV) thus corresponding to the sensor size. Figure 3b shows the orientation of the manufactured part on the magnesium AZ31 substrate. One rectangular layer was built with the dimensions 14 mm * 10 mm with the camera's FOV in the center with dimensions of 10 mm * 6 mm. This way, a maximum number of frames is collected from the center of the scan vectors. During the LPBF process, all parameters except laser power and scan speed were kept constant.

Data acquisition and preprocessing
Data acquisition and preprocessing scripts were implemented using the camera manufacturer's application programming interface for the programming language Python. This was beneficial for subsequent data evaluation through methods of machine learning since these also rely on Python. Figure 4 shows schematically how data acquisition, storage and subsequent evaluation were organized. At the start of the process, the frames were continually captured and the maximum grey value was determined. As soon as the laser spot reached the FOV of the camera, this value quickly shot up. The storage of the images therefore started when the maximum grey value had reached a threshold of 20 for at least 20 ms (at an exposure time of 4 ms this corresponds to five consecutive frames). The frames were then saved at full resolution as a numpy-array data type and were successively put into a list until the grey value dropped below 20 for 20 ms. This was the case when the laser spot moved out of the camera's FOV and the coating process started. Then, the first data processing occurred: for every frame, the pixel coordinates of the maximum grey value were determined which then served as the center for converting the 2048 * 1088 frame to a 100 * 100 frame. The cropping was then done such that the spectral bands' pixel locations were identical for each post-processed frame. Efficient data processing requires manageable amounts of data and so the rest of the image was discarded.
All frames that were captured in one layer were saved after cropping in a so-called pickle-file that served as a container of the Python list. The processing of the frames is usually interrupted as soon as the next layer is built. For the experiments presented here, however, only one layer was built. The following data processing happened only after the build job has finished. At first, the 100 * 100 * 1 frames were dissected into 20 * 20 * 25 frames to sort the different spectral channels. As soon as the samples were investigated and the quantities of interest were determined, each frame could be assigned a label-in this case the surface roughness R z . The result was a Python list where each element was again a list containing the frame as a numpy-array and the label as a float. This list could be used as input for a convolutional neural network.

Measurement of surface roughness
The target value of the investigations was the surface roughness of additively manufactured WE43 alloy. In order to keep ambient conditions as reproducible as possible, only one layer was manufactured while the process was monitored with the hyperspectral camera. Three parameter combinations of laser power and scan speed were selected from literature [8]. One was reported to lead to dense parts and good reproducibility (50 W; 75 mm/s), one to too much energy input (50 W; 40 mm/s), and the final one to too little energy input (20 W; 75 mm/s). The roughness R z was measured according to DIN EN ISO 4288 using a Keyence VK X1050 confocal microscope. The cut-off wavelengths λ C and λ S were set to 2.5 mm and 8 μm, respectively. The measuring section l n was therefore 12.5 mm long. With a FOV of 10 mm * 6 mm, the roughness was measured only along one direction so that most of the measuring section lay within the FOV. Figure 5a shows the confocal image of the first sample and the location of the five measuring sections. The average of the five roughness values was taken as the final value that was used as a label for all the camera   Table 4 shows the surface roughness measured of the different samples. Sorted by parameter set, for each sample, the average measured roughness R z is given along with the number of frames acquired from the hyperspectral camera for this particular sample in brackets. The right column finally displays the mean roughness for the parameter set and the mean absolute error (or deviation) of that set. These values will help evaluate the prediction power of the convolutional neural network used subsequently.

Predicting the surface roughness using a convolutional neural network
The correlation between the surface roughness and the hyperspectral process recordings shall be determined. As laid out in previous sections, the images contain spatial as  This information could potentially contain crucial features for the prediction of the surface roughness. Since these features are unknown, a method from machine learning, a so-called convolutional neural network, is applied. The hyperspectral measurement is done in the high-temperature melt pool environment. Therefore, the temperature has an obvious influence on the hyperspectral measurement. Since the training data is labeled with the roughness, however, the model is trained to extract precisely the features related to roughness, not temperature. Typically, the input data for a CNN are images with one or three color channels. The process recordings in this case, however, entail 25 channels, such that the network topology is adapted accordingly. Therefore, pre-trained networks that have achieved exceedingly high classification accuracies in other areas (e.g. ResNet-models) cannot be applied in this case.
On the one hand, the CNN shall provide a high prediction accuracy while on the other hand it may not require more computation power than was available, both for the training and for the later classification process. That is why an approach with four convolutional layers was chosen (Fig. 6) whose number of channels is doubled in each except for the last layer. The dimension of the convolutional matrix was fixed at 3 * 3. The ReLU-function served as activation function. The input data in the form of the process images entailed a large number of channels (25) but was otherwise very small (20 * 20) which is why no max-pooling layers were applied. To avoid overfitting, a dropout layer was added to the network followed by a fully connected layer before the final output is given. The mean squared error was chosen as a loss function to penalize high deviations between prediction and label during training. As evaluation metric, the mean absolute error was used, so that it could directly be compared to the values in Table 4. To compare the prediction power of different network topologies, a hyperparameter tuning was conducted during which the dropout rate (0.4 and 0.6), the number of perceptrons in the fully connected layer (8,16,32), and the optimizer (Adam and Nadam) were varied. Table 5 shows the results of the training for 10 epochs, respectively, with a batch size of 20 with the corresponding hyperparameters. Of the data available, 70% was used as training, 20% as validation, and the remaining 10% as test data. The test data was used to determine the prediction accuracies shown. For better clarity, the results are visualized in Fig. 7, such that the influence of individual hyperparameters becomes apparent. It can be seen, for instance, that the Nadamoptimizer allows higher prediction accuracies than the Adam optimizer. The smallest mean absolute error and thus the highest prediction accuracy was achieved using the Nadam optimizer, a dropout rate of 0.6 and 16 perceptrons in the fully connected layer. A simplified scheme of this network topology is shown in Fig. 6.
With these hyperparameters, a longer training process of 40 epochs was carried out. To ensure that for each parameter set the same proportion of data went into training, validation and test set, the parameter sets were divided separately. This means that for each parameter set, 70% of the data was allocated to the training, 20% to the validation, and 10% to the test data set. Only after that, the training and validation data was shuffled. The test data was kept separate so that the prediction accuracy could be evaluated for each parameter set individually. This allows to determine whether the CNN really recognizes  Figure 8 shows the evolution of the loss function during the training process for training and validation data. Despite the high dropout rate of 0.6, overfitting can be observed starting at around epoch 22. The network learns the features of the training data "too well". Table 6 shows how accurately the CNN can predict the surface roughness based on the test data. For the complete data set, all three parameter sets were used. The prediction accuracy of the CNN can be expressed by an MAE of 4.1 μm which is much lower than the MAE of the roughness within the data set of 14.3 μm. It should be noted that the results of parameter set A contribute slightly more to this result than the other parameter sets due to sheer number of images in each set. As a next step, the separate test data   Table 6 shows that for parameter set A (50 W; 75 mm/s), the parameter itself is recognized, but the roughness cannot be predicted more accurately than is already implied by the statistical spread within that set. For parameter set B (50 W; 40 mm/s), on the other hand, the roughness can be predicted more accurately than implied by the inherent spread. For this set, there must be features in the hyperspectral data that contain information about the surface roughness. Parameter set C (20 W; 75 mm/s) shows a high spread and the roughness can be predicted even less accurately.
First of all, these results show that the different parameters can be recognized fairly well. Within the different parameter sets, however, the surface roughness R z can be predicted with diverging accuracy when compared to the statistical spread in that set. The spread of the roughness in parameter set A is very small, making a more accurate prediction difficult. Parameter set C consists of a very low laser power, so that too little light was available for a more accurate prediction. Parameter set B, on the other hand, contains sufficient statistical spread as well as enough light so that the prediction is more accurate than the statistical spread. Besides, the largest amount of training data was available for this parameter set.

Conclusion
The aim of the research presented in this article was to investigate the correlation of hyperspectral recordings from the process and part properties, in this case the surface roughness R z . To this end, the area-scan hyperspectral camera was adapted to the process and integrated into a laboratory machine. As a material particularly relevant for biomedical applications, the magnesium alloy WE43 was used in the experiments. The most relevant powder properties like particle size distribution, sphericity and moisture content were determined to ensure a reliable process. The area-scan hyperspectral camera is a compromise between spatial and spectral resolution and as such generates large amounts of data. The data processing methodology developed in this work has proven to enable efficient data handling nonetheless by extracting only the relevant parts. In order to establish the link between the multi-channel data and the surface roughness, a convolutional neural network was adapted to the input and a hyperparameter tuning was carried out. During training of the final network topology for 40 epochs, overfitting was observed despite an integrated dropout layer. The overall prediction accuracy over the entire data including three different parameter sets corresponds to an MAE of 4.1 μm compared to an MAE of the actual roughness of 14.3 μm. The MAE of the prediction within a parameter set, however, is not always smaller than that of the actual roughness in that set.
Overall, the results suggest that hyperspectral data does provide crucial information about the LPBF process and that hyperspectral imaging could potentially help establish on-line product qualification. It has yet to be investigated, however, whether other part properties apart from the surface roughness and spatially resolved process defects can be predicted as well. This will also show whether the compromises in terms of spatial, spectral and temporal resolution that go along with this kind of sensor are justified. This should be the subject of future research in this area.
Author contribution N. Gerdes conceived of the study, formulated its design, coordinated the conduct of the study including data collection, designed the convolutional neural network, and interpreted its output. He also drafted the manuscript with support from C. Hoff and L. Overmeyer. All authors discussed the results and contributed to the final manuscript.
Funding Open Access funding enabled and organized by Projekt DEAL. The authors gratefully acknowledge the funding by the German Research Foundation (DFG) within the priority program (SPP) 2122 "Materials for Additive Manufacturing (MATframe)".
Availability of data and material The data sets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Code availability
The code generated during the current study is available from the corresponding author on reasonable request.

Declarations
Ethics approval Not applicable.

Consent to participate Not applicable.
Consent to publish Not applicable.

Competing interests
The authors declare no competing interests.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons. org/licenses/by/4.0/.