Improved quality assessment of colour surfaces for additive manufacturing based on image entropy

A reliable automatic visual quality assessment of 3D-printed surfaces is one of the key issues related to computer and machine vision in the Industry 4.0 era. The colour-independent method based on image entropy proposed in the paper makes it possible to detect and identify some typical problems visible on the surfaces of objects obtained by additive manufacturing. Depending on the quality factor, some of such 3D printing failures may be corrected during the printing process or the operation can be aborted to save time and filament. Since the surface quality of 3D-printed objects may be related to some mechanical or physical properties of obtained objects, its fast and reliable evaluation may also be helpful during the quality monitoring procedures. The method presented in the paper utilizes the assumption of the increase of image entropy for irregularly distorted 3D-printed surfaces. Nevertheless, because of the local nature of distortions, the direct application of the global entropy does not lead to satisfactory results of automatic surface quality assessment. Therefore, the extended method, based on the combination of the local image entropy and its variance with additional colour adjustment, is proposed in the paper, leading to the proper classification of 78 samples used during the experimental verification of the proposed approach.


Introduction
Additive manufacturing has become one of the most dynamically developing technologies related to Industry 4.0 in recent years. The use of relatively cheap and widely available 3D printers makes the rapid prototyping and on-demand production of various unique elements and details possible. Due to a long time necessary for printing of some larger objects as well as technical imperfections of some low-cost devices, one of the relevant issues is a reliable quality assessment of the printed elements, especially their surfaces.
Automatic detections of some distortions during the printing process make it possible to perform potential corrections in some cases, or for more significant malformations, the decision about aborting the manufacturing can be made. In such online applications, the visual quality assessment should be made preferably in controlled lighting conditions by the analysis of images captured by the cameras observing the side surface of the printed element. Such an assumption makes it possible to check the regularity of each layer of the filament, and therefore, the amount and size of detected deficiencies may be estimated.
In general manner, 3D printing methods can be divided into four major groups: stereolithography, ink-jet printing, selective laser sintering and fused deposition modelling (FDM). The last one, utilizing layer deposition of a thermoplastic material extruded from a software-controlled nozzle, is the most popular, and therefore, the methods presented in this paper will be focused on the FDM. Since the most popular thermoplastic filaments are polylactide (PLA) and acrylonitrile butadiene styrene (ABS), all the samples used in experiments have been prepared using various colours of such materials.
Automatic visual quality assessment of the 3D-printed surfaces is a relatively new, yet important, area of computer vision research. Although it constitutes a major technical challenge, it also leads to some new possibilities of 1 3 interdisciplinary integration of various kinds of information. The detection of some cracks or larger distortions visible on the printed surface may be considered as an indication of weak mechanical properties of the manufactured element. On the other hand, some internal contaminations which cannot be detected using purely visual analysis might be assessed using similar methods combined with electromagnetic (e.g. terahertz based) nondestructive testing methods, successfully applied to evaluation of optical properties for various filaments [1]. Some possibilities of applications of X-ray computed tomography and ultrasonic imaging for detection of embedded defects and effects of altered printing orientation have been discussed by Zeltmann et al. [2].
As stated by Straub [3], the decision system which uses the results of the automatic quality assessment is dependent on the hardware possibilities. In the basic version, only the decision about stopping or continuing the printing process can be made. In more advanced systems, some additional actions can be made, e.g. re-application of filament to detected places (sometimes even before the start of the deposition of the next layer, assuming the observation from the top) or milling the areas having too much of the deposited material.
The rest of this paper is organized as follows: Sect. 2.1 focuses on the known applications of image analysis for monitoring the 3D printing process, in Sect. 2, some previous attempts to quality assessment have been presented together with discussion of initial experiments and Sect. 3 is related to the description of the proposed approach, whereas the next section contains the analysis of obtained experimental results and is followed by conclusions.

Overview of machine vision in monitoring of 3D printing
One of the first attempts to visual monitoring of the 3D printing has been the concept of process signatures applied to fused deposition of ceramics proposed by Fang et al. [4,5]. This approach focuses only on defect detection using simplified texture analysis by comparing the image of the printed sample with the expected virtual one. Another idea presented by Cheng and Jafari [6] is based on the monitoring of top surface during printing to prevent the presence of voids. It is based on the process dynamics models, and the correlation between adjacent layers for the reconstructed 3D road shapes using shape from profile makes the identification of a single type of contamination possible (caused by under-or overfilling). Another application of vision-based methods for fault detection has been proposed by Szkilnyk, Hughes and Surgenor [7], which makes it possible to recognize some defects, e.g. jams, however only those previously defined. A comparison of some other machine vision methods used for fault detection has been provided by Chauhan and Surgenor [8,9], whereas Straub [10] has described the system dedicated to initial image analysis used for comparison of the inprocess object with the final one. However, considering the pixel-by-pixel comparison used by the Author, an accurate calibration of the camera and the printing device is necessary to obtain good results. The proposed system has been found very sensitive to environmental and even small camera position changes. Some alignment issues have been further described in the paper [11].
An interesting solution for quality assessment of 3D-printed electronic products has been proposed by Tourlokis [12]. It has been applied to 3D ink-jet printing and focuses on the analysis of droplets using neural networks. However, considering the required training for the network and the specific application, the universality of this method is considered as limited.
Another initial attempt to visual inspection of 3D printing has been presented by Makagonov, Blinova and Bezukladnikov [13]. Detection of singular points based on the use of functions available in OpenCV library made it possible to match properly the images of 3D-printed parts with their models shown on reference images. Nevertheless, the quality assessment of the printed surfaces has not been applied. Holzmond and Li [14] have proposed the use of 3D image correlation system for monitoring the surface geometry of the printed parts, which requires the reconstruction of 3D images using two cameras and further comparison with a 3D CAD model. Scime and Beuth [15] have used the filter banks and k-means unsupervised clustering for training and further anomaly detection in parts obtained from laser powder bed fusion 3D printers.
One of the recent most relevant achievements is the application of machine vision to support the multimaterial ink-jet 3D printing. This project, presented at SIGGRAPH 2015 as MultiFab [16], assumes self-calibration of printheads and closed visual feedback loop for correcting the printed objects based on the 3D scanning. Nevertheless, the machine vision module of the system utilizes optical coherence tomography (OCT) with the use of Michelson interferometer, beam splitter and light polarizers and therefore this solution cannot be considered as really low cost (the total price is about $7,000). Detection of corrupted areas with missing filament is made with the use of depth map analysis acquired after additional calibration of optical parts and double scanning the surface of the empty platform. The additional mentioned limitation of the OCT scanner applied to quality monitoring purposes is its inability of detection of overhangs or undercuts. Another application of the OCT to in situ process monitoring for selective laser sintering 3D printers has been recently proposed by Gardner et al. [17].
An interesting application of machine learning for quality monitoring of the 3D prints has been examined by Delli and Chang [18]. However, the main drawback of this approach, utilizing the previously trained SVM model, is its inability to assess the defects on the vertical plane as only the topview camera has been used. Another issue is the necessity of pausing the printing process when taking photographs of unfinished parts.

Quality assessment of 3D prints
Considering the approaches discussed in Sect. 2.1, the application of image analysis for quality evaluation of the 3D prints seems to be a strongly desirable solution, although most of the researchers limit their solutions to process monitoring or fault detection. To face this challenge, we have decided to examine the possibilities of applying texture analysis methods based on the features extracted from the greylevel co-occurrence matrix (GLCM). Assuming the visibility of consecutive layers of the filament (which should be regular for high-quality FDM prints) in controlled lighting conditions, we have decided to use both photographs and scanned images for verification purposes. The first experiments made it possible to distinguish both high-quality surfaces from the low-quality prints as well as photographs from scans [19,20]. Nevertheless, it requires the time-consuming calculations of series of GLCMs and their features: homogeneity for quality evaluation and correlation for the detection of the image type.
Another investigated approach is based on the use of some full-reference image quality assessment (IQA) methods, such as structural similarity (SSIM) and its modifications as well as feature similarity (FSIM) [21,22]. Since the use of full-reference methods, based on the comparison with the reference image, would require a calibrated model of the 3D-printed surface, we have proposed a solution based on the division of images into regions (equivalent to blocks, avoiding the use of the word "windows" due to a possible misunderstanding caused by the use of "sliding windows" in convolutional filters as well as similarity-based IQA methods, such as aforementioned SSIM or FSIM).
Consequently, it is possible to calculate the mutual similarity indexes between the regions whose low values indicate low quality of the surfaces due to irregularity of possible contaminations. A regular distribution of such distortions in practical applications appears pretty unlikely; however, in such cases, a change of the number of regions may be used to solve any issues arising from potentially too high similarity values. Assuming that the same local contaminations are visible in the central parts of 16 regions obtained using the 4 × 4 grid, the use of nine or 25 regions (3 × 3 or 5 × 5 grids, respectively) causes their shifting according to each other when those regions are compared.
A similar idea of comparisons, although for some randomly chosen fragments of the 3D-printed surfaces, has also been proposed for the local cross-correlation [23]. One of the advantages of this approach is the decrease in the computational cost. However, the application of this method for a higher number of samples with different colours would require some modifications as well as the choice of appropriate parameters and additional adjustments. One of the drawbacks of this approach is also the necessity of using high-resolution scans, which limits its potential applicability in real-time assessment of images acquired by cameras.
The initial attempt to the use of entropy has been presented in the paper [24]. The idea of its application is based on the assumption of low image entropy for regular patterns, which can be observed for high-quality flat 3D-printed surfaces. However, the calculation of a single entropy value for the whole image does not lead to satisfactory results; therefore, its local values should be computed as well. In the initial experiments, some different colour-to-greyscale conversions have been examined and the best results have been obtained using the well-known ITU-R recommendation BT.601-7 [25], which is equivalent to the use of rgb2gray function in MATLAB environment. Nevertheless, the influence of the chosen conversion method is not significant and the classification of samples into low-and high-quality ones could be possible using independent thresholds for each colour.
A further extension of this approach has been described in the paper [26]. Since a strong influence of the filament's colour on the entropy of the scanned image can be observed, a colour-independent quality evaluation should be based on the combination of the local entropy values and their variances calculated for the RGB and HSV colour models. The detailed results obtained for 18 high-and low-quality samples obtained for five colours of PLA filaments can be found in our earlier publication [26], which is the basic point for this paper.
As mentioned above, all the experiments and results presented in the discussed papers have been obtained for a relatively small number of samples with limited number of colours. Therefore, to provide a highly efficient colourindependent method, some more comprehensive experiments have been made using not only five colours of the PLA samples but also some ABS filaments (five additional colours). Additionally, the newer 3D-printed samples contain varying amount of contaminations and hence they are divided into four groups according to subjective assessment as high-, moderately high, moderately low and low-quality surfaces. Those groups result from the analysis of subjective quality scores provided by the members of our scientific group. During the subjective assessment, both sides of each sample have been considered, which are stored as two separate images in the data set used in calculations. Samples containing visible distortions, forced by changing the temperature or speed of filament's feeding using controlled stepper motors, have been subjectively assessed as low or moderately low quality, depending on the amount and density of contaminations. It is worth to note that some of the samples containing small cracks, actually being the result of lack of adhesion between the neighbouring layers, although without significant other quality issues, might have been assessed as moderately high or even high quality.
In some applications, the presence of cracks may be considered as enough to classify such samples as low-quality ones, e.g. due to the loss of mechanical properties. Nevertheless, since cracks might be possible to correct after manufacturing, it has been assumed that their role is a bit different from aesthetic point of view than some other quality issues caused by overfilling or underfilling. Hence, the eight possible types of surfaces (four quality scores, each with or without cracks) have been presented in the subsequent plots. All the samples have been produced in our laboratory-the reference samples have been obtained using Da Vinci 1.0 Pro 3-in-1 3D printer shown in Fig. 1, whereas for the contaminated samples, two older devices, namely RepRap Pro Ormerod2 and Prusa i3, have been used.
The idea of using a single measure of image quality comes from general-purpose image quality assessment methods, e.g. structural similarity (SSIM) [27] and its modifications, such as MS-SSIM or FSIM [28]. Actually, even the hybrid metrics based on multimetric fusion [29,30] applied for general-purpose IQA produce a single output value, which is easy for the interpretation. Usually, image quality assessment problem is not considered as a typical classification task, when the division into two classes should only be made. Since one may expect the presence of images of different quality, also moderate, some databases, e.g. TID, LIVE, CSIQ, etc. are available, which contain the original and distorted images assessed by numerous human observers with aggregated subjective scores expressed as mean opinion scores (MOS) or differential MOS values [31]. Unfortunately, there are no such data sets containing the images of 3D-printed surfaces, as well as generally accepted methods of automatic evaluation of their quality. Hence, to let the user interpret the single measure, we have decided not to use the typical classification approach for the four features.
In other words, our goal is not classification but automatically computing the quality score, which could be used for classification of samples containing the additional information, especially if the quality is very high or low, or is quite close to the threshold. In future applications, it could be useful, when the decision of aborting or continuing the manufacturing process should be made. Assuming that the quality of the printed samples is not always perfect, different decisions could be made, depending on how much distorted the manufactured surface is.

Discussion of the proposed method
Some of the main challenges of the automatic quality evaluation of object surfaces, especially being obtained using lowcost additive manufacturing devices, are the colour independence of the assessment results, relatively low required computational cost and robustness to lighting conditions. Considering the complexity issues and demands, there is no need to provide the real-time analysis of images because of relatively low printing speed. Therefore, the analysis of a few frames per second should be enough and nowadays it can be obtained using many available hardware solutions. Ensuring the uniform illumination is possible by the use of dedicated illuminators, typically applied in industrial applications based on machine vision, especially assuming the closed housing of the 3D printer.
Therefore, in our research, we have focused on colour independence of the assessment method using the images of surfaces of the 3D-printed flat plates, obtained using the flatbed scanner at 1200 dpi resolution to ensure the visibility of individual layers of the filament. As mentioned, for the Fig. 1 Photograph of the Da Vinci 1.0 Pro 3-in-1 device used mainly for additive manufacturing of the reference samples made in our laboratory verification of the developed algorithm, a larger data set of images has been prepared with the use of various colours of PLA and ABS filaments, forcing the presence of differing amount of distortions. The resolution of such obtained images is 1600 × 1600 pixels, which is equivalent to 35 × 35 mm 2 , considering the physical size of the samples. The thickness of the samples is about 4 mm, whereas the height of each layer varies from 0.3 to 0.35 mm depending on the printer and the size of the nozzle. For such assumptions, each layer visible on the 3D-printed surface is represented by several pixels on the image. As the obtained images are square, all the resulting blocks are also square, and therefore, their size is 200 × 200 pixels for 64 blocks and 100 × 100 pixels for 256 blocks, respectively.
Since the presence of contaminations visible on the 3D-printed surface increases the local image entropy, not only the absolute entropy values but also their variance can be useful for quality assessment purposes. One of the reasons is the dependence of the entropy on the overall brightness of images, which should be balanced to ensure the colour independence of the final quality metric. For a single channel of the image (or after its conversion to greyscale), the entropy, considered as a statistical measure of randomness, can be calculated as where the vector p contains the histogram counts calculated for the N bins (typically 256).
An illustration of the usefulness of the entropy for the detection of local contaminations and irregularities of the 3D-printed surfaces is shown in Fig. 2, where its increase for highly corrupted regions is well visible. It is worth noting that the average entropy calculated for all 16 parts, equal to 4.8185, is significantly lower than its global value calculated for the whole image. Nevertheless, the results obtained for various filaments differ noticeably, and therefore, the proper classification into high-and low-quality 3D prints might be troublesome without the prior knowledge of the filament's colour as shown in Fig. 3. Some more detailed results for the first 18 samples have also been presented in [24].
As mentioned earlier, the direct use of entropy might be combined with the additional factor, namely the variance of local entropy, as proposed in the conference paper [26]. Since low entropy is typical for high-quality 3D prints and high variance of the local entropy indicates low quality, it has been assumed that their combination should provide a better classification, particularly when expressed in logarithmic scale. Therefore, this mean-variance (MV) formula can be expressed as  value Entropy high quality samples moderately high quality samples moderately low quality samples low quality samples high quality samples with cracks moderately high quality samples with cracks moderately low quality samples with cracks low quality samples with cracks where the EV factors are the products of the average local entropy and its variance calculated in RGB and HSV colour spaces. Local entropy values used in the above formula for the RGB colour space are calculated for the average of three colour channels, whereas for the HSV colour space, only the hue component is used in calculations. The number of image fragments can be a subject of further optimization. As illustrated in Fig. 4, although this approach makes the proper classification of the first 18 PLA samples possible, its application to the other colours of ABS filaments does not lead to satisfactory results. Considering moderately low quality as low and moderately high as high, respectively, the use of the proposed threshold equal to −7 is generally correct for the brown samples. Nevertheless, it is not valid for any of the other colours of the ABS 3D prints used in the experiments. Since one of the potential reasons of lower performance may be the presence of some less visible distortions, especially on moderately low and moderately high quality surfaces, a quite obvious extension of the method might be the increase in the number of image blocks into 64 or 256. The results obtained for these assumptions are shown in Figs. 5 and 6, respectively. Analysing the latter plot, slightly better results have been obtained for yellow (samples no. 49-60) and salmon colour filaments (samples no. 29-40), making it possible to use the same threshold value as for the first 18 PLA samples. It should be, however, noted that the MV values of the most of the samples increase due to higher number of blocks, causing mainly higher variance of local entropy values, and therefore, the most appropriate threshold value should also be increased for most colours. This  The results obtained using the proposed approach are shown in Fig. 8, whereas their equivalents achieved assuming the division of images into 64 blocks are illustrated in Fig. 9. As can be easily noticed, the appropriate threshold for the proposed method is equal to −6 . Unfortunately, the use of 64 larger blocks does not lead to satisfactory classification due to some issues for yellow samples (as shown in Fig. 9) and even worse results have been achieved for 16 blocks. The attempts to optimize the coefficients of formula (3) have not led to the expected solution in both those cases, and therefore the division into 256 smaller blocks turned out to be necessary. To present a visual comparison of the quality of some selected 3D-printed objects, sample surface images together with MV AVGcol values are shown in Fig. 10. The additional illustration of samples of the same colour with varying quality is shown in Fig. 11.

Discussion of results
During the experiments, the final classification into only two classes, representing high-and low-quality samples, has been assumed, although subjective scores have been divided into four groups. The reason for introducing two Fig. 8 Results obtained for the proposed metric and the division into 256 fragments with a colour-independent classification of 3D-printed surfaces according to their quality Fig. 9 Results obtained for the proposed metric and the division into 64 fragments additional classes is related to the interpretation of results. Since the accuracy of classification is not always 100%, particularly during initial research, the main focus has been set on the proper classification of samples subjectively assessed as "high" and "low" quality as the initial requirement, further improving the classification accuracy for "moderately high" and "moderately low" quality samples. One of the assumptions of our future research is the introduction of a nearly continuous subjective quality scale. However, it would require the development of a bigger database containing also numerous subjective scores delivered by a relatively large group of human observers, similarly as in general-purpose image quality assessment data sets (e.g. LIVE or TID).
An additional verification of the necessity of all four main features (local entropy values and their variances in RGB colour space as well as for the hue component) has been made by the appropriate simplifications of formulas 2 and 4. The results obtained using only the MV metric calculated for the RGB colour space are shown in Fig. 12, whereas those achieved for the hue component are illustrated in Fig. 13. Both plots confirm the decrease in the classification accuracy reached using such simplified combinations of features. Additionally, a set of values of some popular classification metrics obtained for various cases is presented in Table 1, considering high-quality  samples as positives and low-quality ones as negatives. Hence, F-Measure and accuracy may be calculated using the following formulas: Analysing the presented results, it may be noticed that in some cases for the simplified methods, where the classification accuracy is not perfect, improperly classified samples belong to the "moderately low" and "moderately high" quality groups. The decrease in the number of regions to 64 causes the improper classification of two yellow samples  The extended colour-independent method of image quality evaluation of the 3D-printed surfaces based on image entropy proposed in the paper makes it possible to assess different samples reliably, regardless of the used filaments. Although the presented method has been verified for 78 flat surfaces manufactured from two types of materials (PLA and ABS), the application of the presented approach can be easily extended also for some other shapes (e.g. convex surfaces) since there are no assumptions related to the presence of straight lines in contrast to, e.g., the use of Hough transform or local similarity metrics [21,22]. Since the proposed metric is based on image entropy, which is independent on the orientation of the lines (as, e.g., the assessment based on Hough transform), it may be applied for curved surfaces similarly as for the flat ones (with necessary adjustment of the classification threshold). Some photographs of additional samples used during verification and further experiments are shown in Fig. 15 together with obtained results.
Another generalization may be related to the automatic choice of the size of the analysed block, especially for online quality monitoring purposes. Considering the presented results, a proper classification is possible for the blocks containing about ten visible layers of the printed material, being a compromise between the computational complexity and required classification accuracy.
Since our approach is not the "standard" classification, it is not based on the division into training and testing data sets. As our goal is to find the combination of features based on entropy ensuring a proper thresholding between the classes, the proposed method does not require the training of any classifiers. Nevertheless, a closed-form formula with an appropriate threshold presented in the paper makes it possible to obtain the correct classification of the 78 samples used in experiments as well as convenient interpretation of obtained results by the users. The classification has been made by using formula (4) and setting the optimal threshold with the minimum distances to the two nearest values obtained for high-and low-quality samples and maximum accuracy calculated according to formula (6).

Concluding remarks
Considering the possible application of machine learning methods, including growing popularity of deep neural networks, their use would require a much higher number of samples. Since manufacturing of the individual 3D-printed objects is a quite time-consuming task, the use of less than 100 images has been assumed. Therefore, we have decided not to use typical machine learning methods, as well as deep CNN-based approach. Since our goal is to develop a colour-independent quality assessment method, we have not considered each colour as a separate class, bringing all samples together. In comparison with many general image quality assessment databases, containing typically less than 1000 images, where the diversity of contents, e.g. shapes, textures, colours, etc., is much bigger, it has been assumed that the number of 78 samples representing only 3D-printed surfaces is enough, being also higher than in some of medical image databases where gathering of data might also be troublesome.
Although our analysis has been made off-line after manufacturing, we have used relatively small flat samples in our experiments, assuming that they may be considered as some fragments of a bigger object. Therefore, assuming an appropriate location of a camera, the proposed method may be applied for quality monitoring during the 3D printing process as well.
One of the directions of our future work will be concentrated on the further improvement of the proposed method towards its robustness to nonuniform lighting conditions. Such extension would be desired, especially for the online quality monitoring of the 3D-printed objects in devices without closed casings. Another interesting challenges are the development of the quality assessment method correlated with subjective perception of observed degradations as well as the utilization of stereo-vision and depth data.
Another direction of our further research is the combination of the method proposed in the paper with some additional features which were investigated in earlier papers, e.g. based on texture analysis [19], Hough transform [21] or some image quality assessment methods based on the mutual structural similarity [22]. Such combination of features might be useful, especially for uncontrolled lighting conditions improving the robustness of the method.

Compliance with ethical standards
Conflict of interest The authors declare that they have no conflict of interest.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creat iveco mmons .org/licen ses/by/4.0/.