1 Introduction

A crucial issue in manufacturing, when accurate inspection and rapid production are necessary, is the designation and inspection of machined metal surfaces. The automatic inspection and detection of defects using image processing is an area of machine vision that is widely adopted in many industrial fields. It is used for high-throughput quality control in production systems, such as the detection of flaws on manufactured surfaces of automobiles or mobile phones [1, 2]. The idea is to design autonomous devices that automatically examine and improve the performance of traditional inspection systems that depend heavily on human inspectors during the lapping process for a mold product surface. This task is usually repetitive and labor intensive, and it might also involve exposure to radiation, high noise levels, metal dust, and chemical environments, leading to health problems such as musculoskeletal and lung diseases. Therefore, robotic finishing for automation is highly necessary from the perspective of productivity, cost, time, and uniformity of surface quality [3, 4].

The finishing performance of a surface depends on the dimensional characteristics of the surface irregularities. These characteristics vary considerably according to the surface material and the manufacturing process.

Conventionally, surface roughness and waviness are measured using a stylus, laser scanners, etc. However, these methods are rather slow and time consuming since they measure the surface profile at every point on the surface. In practice, many circumstances exist in which such detailed information about the metal surface is unnecessary. Moreover, the methods create limitations to the setup. Because of these problems, it is difficult to apply a stylus to real industrial sites or robotic finishing [4,5,6].

Alternatively, non-contact optical metrology techniques can be employed to overcome these problems. These techniques include confocal microscopy; various interferometry methods; optical profiling; and methods using reflection, scattering, and speckle patterns [4,5,6,7,8]. Although many methods are available, the use of these techniques is limited at industrial sites because of the working environment and special issues for measurement. Hence, a surface inspection is required that is high speed, robust, low cost, and applicable to both industrial sites and robot finishing [4, 9,10,11,12].

There are two types of roughness in the lapping mold surface. The purpose of this research is not only to detect the machining induced tool marks that are regularly distributed, but the unexpectedly appeared scratches that are very thin and irregular type. This paper compares two major image-based surface techniques. First, machined surfaces are evaluated using tool mark detection and classification of the surface state based on the characteristics of the different reflected light according to the residual tool mark. The second proposed method is inspecting the entire machined surface at once and detecting scratches from the extracted normal map image based on a photometric stereo.

Captured near-field contrast images right after the reflected specular light made by an individual point light provide highly sensitive and reliable image data so that they can be processed through grayscale.

2 Inspection Based on Reflection Property

2.1 Reflection Property of Machined Surface [13]

After machining, multiple tool marks are created on the metal surface by machining tools performing milling and grinding. When a point light source is used to illuminate the surface, reflected light is generated in a direction perpendicular to the tool mark. The reflected light is indicated by the yellow arrows in Fig. 1 and is called a specular highlight.

Fig. 1
figure 1

Schematic diagram of the machined metal surface inspection

Full size image

The reflected light can be expressed as Eq. (1) in relation to the tool mark and the normal vector of the surface:

$$ \widehat{{r_{h} }} = \hat{s} \times \hat{n} $$
(1)

where \( \widehat{{r_{h} }} \): the specular highlight by tool mark, \( \hat{s} \): the tool mark direction of surface, \( \hat{n} \): the surface normal.

Equation (2) is satisfied when the intensity is maximum:

$$ \widehat{{n^{\prime}}} \cdot (\hat{s} \times \hat{n}) = \widehat{{n^{\prime}}} \times \widehat{{r_{h} }} = 0 $$
(2)

where \( \widehat{{n^{\prime}}} \): the bisector between camera and light source.

2.2 Reflection of the Milled Surfaces

In the case of milled surfaces, the highlight is made by gathering the reflected light in a small groove, as magnified by the red circle in Fig. 2.

Fig. 2
figure 2

Specular highlight in the milled surface

Full size image

The milling tool pattern is circular and has various diameters. Because of this, different reflected light patterns are created depending on the tool diameter, as shown in Fig. 3.

Fig. 3
figure 3

Relationship between milling tool diameter and reflected light

Full size image

Using the tool mark radius from a point P, the direction vector of the milling tool mark is given by Eq. (3):

$$ \hat{s} = \frac{1}{r}\left( {\begin{array}{*{20}c} {P_{x} } \\ {\left(r^{2} - P_{x}^{2} \right)^{1/2} } \\ 0 \\ \end{array} } \right),\quad P_{x} \in [ - r \cdots r] $$
(3)

where \( r \): the radius of milling tool, \( P \): a point on the milled surface, \( P_{x} \): the point depending on the x-coordinate.

The below equation is derived by substituting Eq. (3) for the maximum intensity condition:

$$ \begin{aligned} & \frac{1}{r}\widehat{{n^{\prime}}} \cdot \left\{ {\left( {\begin{array}{*{20}c} 0 \\ 0 \\ 1 \\ \end{array} } \right) \times \left[ {\begin{array}{*{20}c} {P_{x} } \\ {\left(r^{2} - P_{x}^{2} \right)^{{\frac{1}{2}}} } \\ 0 \\ \end{array} } \right]} \right\} = 0 \\ & \quad \Leftrightarrow \frac{1}{r}\left[ {\widehat{{n_{x}^{\prime } }}(r^{2} - P_{x}^{2} )^{{\frac{1}{2}}} + \widehat{{n_{y}^{\prime } }}P_{x} } \right] = 0 \\ \end{aligned} $$

From the reflection property and maximum intensity condition, the relationship between the milling tool radius and the reflected light is derived at \( P_{x} \) on the surface, as shown in Eq. (4). It can be seen in Eq. (4) that the larger the diameter of the milling tool, the larger the \( P_{x} \). This reduces the range in which the reflected light occurs:

$$ \begin{aligned} & P_{x} = \frac{{rn_{x}^{\prime } }}{{(n_{y}^{\prime } + n_{x}^{\prime } )^{1/2} }}\quad if\,n_{x}^{\prime } \cdot n_{y}^{\prime } < 0 \\ & P_{x} = - \frac{{rn_{x}^{\prime } }}{{(n_{y}^{\prime } + n_{x}^{\prime } )^{{\frac{1}{2}}} }}\quad if\,n_{x}^{\prime } \cdot n_{y}^{\prime } > 0 \\ \end{aligned} $$
(4)

3 Design of the Surface Inspection System

3.1 System Overview

There are many types of surface textures in machined metal parts. These textures can be attributed to one of the following causes:

  • milling tool marks

  • flaws produced during grinding

  • material defects such as cracks

  • chatter marks due to tool vibration

  • scratches due to processing and handling

In order to meet all requirements, the system mainly includes a camera and an LED light source. There are limitations to using laser measuring equipment and precision measuring equipment in real environment conditions [14]. Therefore, the inspection system is tested in a dark space to minimize the effects of external noises. Figure 4 shows the setup for the inspection.

Fig. 4
figure 4

Setup for surface inspection

Full size image
  1. a.

    Camera A SONY RX10 m3 mirrorless camera with built-in CMOS sensor. It runs an ISO sensitivity of 100–12,800 with maximum resolution 20.1 (5472 × 3648). For surface inspection, the maximum ISO sensitivity is used and the distance of the camera zoom is set at 83 mm. The actual resolution used is about 0.02 (mm/pixel).

  2. b.

    Light source 5 mm white LED [luminous intensity: 900 (mcd)] with vertical incidence angle of LED − 45° and horizontal incidence angle of LED − 0° to 180°.

As shown in Fig. 4, the camera is fixed vertically to the specimen, and 19 images are taken from 0° to 180° at intervals of 10° as the workpiece rotates. The light source obtains the image when moving only in the horizontal direction while being fixed at 45° in the vertical direction. As the line light source scans the surface counter-clockwise, the nearfield contrast appears distinctly right after the specular reflected light. The camera captures the images with 50 frame speed.

3.2 Machined Metal Surfaces

For this inspection, four types of machined surfaces are prepared (Fig. 5) which have a size of 50 × 50 × 20 mm3 The material is KP4M.

Fig. 5
figure 5

Machined metal surfaces: a milled surface by a face mill with diameters 50, 80, 100, and 125 mm; the shape of the tool mark is a circular arc; b lapped surface by the turbo-lap linear which has grit sizes #600, #800, and #1200; the shape of the tool mark is straight; c ground surface by grinding machine and hand grinder

Full size image

A total of nine specimens are used for statistical analysis. For these nine surfaces, surfaces were each machined by face mill, a turbo lap linear, a grinding machine, and a hand grinding.

The hand-ground surface is ground three times in the direction perpendicular to the arc of the milling tool mark. The machined surface is ground until the tool mark is not visible during a visual check. In the case of the hand grinder, the tool mark was removed according to a visual inspection at the third grinder.

3.3 Results of Surface Inspection According to Different Property

3.3.1 Milled Surface

The reflected light is perpendicular to the milling tool mark, as shown in Fig. 1. The white line in the image shown in Fig. 6 is a specular highlight or reflected light. As the diameter of the face mill increases, the range of reflected light becomes narrower (Table 1). For a 50 mm face-milled specimen (Fig. 6a), the tool passed twice through the surface because the specimen size and diameter were the same. Thus, reflected light due to two tool marks appears. In the case of 125 mm (Fig. 6d), the depth of the tool mark is uneven due to chatter vibration during machining. Except for the 50 mm case, the overall intensity is dark. However, the intensity in the case of 125 mm is higher than in the 80 mm (Fig. 6b) and 100 mm (Fig. 6c) cases due to chatter marks. The surface roughnesses Ra and Rz are indicated in Table 1.

Fig. 6
figure 6

Milled surfaces images by four types of face mill under 90° of angle light: a face mill 50 mm, b face mill 80 mm, c face mill 100 mm, d face mill 125 mm

Full size image
Table 1 Reflected light range of the milled surface
Full size table

3.3.2 Lapped and Ground Surface

As seen in Table 2, regardless of the grit size of the tool, reflected light by lapping occurs only in the picture when the light source position is 40°–140°.

Table 2 Reflected light range of the lapped and grinded surfaces
Full size table

In addition, the shape of the reflected light by lapping and grinding is linear and wider than the reflected light from the milling tool mark since milling and grinding are performed manually. In the case of #600 (Fig. 7a1–a3), there are two types of reflected light. One is by lapping, and the other is by a milling tool mark. It is possible to check whether the milling tool mark has been removed with the raw image. The total intensity of #600 is lower than in the case of #800 (Fig. 7b) and #1200 (Fig. 7c), 3because the surface roughnesses Ra and Rz improve from #600 to #1200. As the lapping step proceeds, the surface becomes like a mirror surface. The intensity shape of the ground surface is brighter than the lapped surface and does not spread. The overall intensity of the ground surface is relatively dark, like in the milling cases.

Fig. 7
figure 7

Shape of reflected light by lapped and ground surface: a1–a3 lapped by #600, angle of light source respectively 30°, 90°, 150°; b lapped by #800, light source 90°; c lapped by #1200, light source 90°; d plane grinder, light source 90°

Full size image

4 Surface Inspection Algorithm

4.1 Characterization of the Machined Surfaces

When the raw images and surfaces are examined normally, the intensity differs depending on the machining process. Hence, the image processing in MATLAB was adopted to verify the relationship between the intensities seen with the naked eye and those of the image.

As determined by a histogram using 19 images of each machined surface, the gray level with the maximum number of pixels was seen to be different. This dominant gray level or intensity is shown in Fig. 8.

Fig. 8
figure 8

Histogram of gray scale of the machined surface state

Full size image

The milled surface inspections showed a similar gray level for all 19 images, regardless of the diameter of the face mill. This means that the reflected light does not affect the intensity of the entire surface.

The lapped and ground surfaces have a maximum value of 255 gray levels in the region where the reflected light occurs. This means that the intensity of the reflected light dominates the entire intensity. Therefore, this result implies that the intensity and the reflected light indicate surface characteristics that distinguish the machined surface state. However, for lapping, the intensity of #800 is revealed to be higher than that of #1200, which is caused by scratches during the lapping process.

The mean and standard deviation values can represent the intensity in image processing. Figure 9 shows the mean and standard deviation of eight images outside this range. It is confirmed that the intensities of the individual surfaces are distinguished within the range of the histogram without being influenced by the reflected light.

Fig. 9
figure 9

Mean and standard deviation of gray scale corresponds to the intensity of the machined surface state

Full size image

As in the case of the three brown circle areas, surfaces made by milling, grinding, and lapping can be distinguished by the intensity difference.

The mean and standard deviation values of the milled surface occur in one area, except for 125 mm of the three milling cases, since chatter marks exist. In the case of plane grinding, the area of the circles is also very small, because the reflected light forms narrow and uniform on the entire surface.

For the lapping process, the intensities of #800 becomes brighter than those of #1200, as shown in the histogram.

4.2 Determination of the Machined Surface State Based on Intensity

Classification and statistical analyses with the 30 milling, plane grinding, and hand lapping surfaces were carried out. The machined surface state can be determined using the intensity of the surface image.

The T test and ANOVA results for the mean and standard deviation data for the intensity of the machined surfaces in Table 3, which show that the milled and two ground surfaces are independent because the T-test shows that the P-value is less than 0.05, which is clearly distinguished in the Fig. 10. On the other hand, manually hand lapped surfaces after milling machining show much wider distribution in the figure, of which standard deviation is much larger than 0.05.

Table 3 Statistical analysis results of machined surface state
Full size table
Fig. 10
figure 10

Confidence ellipse of the machined surfaces

Full size image

4.3 Quantification of Machined Surface Condition

To quantify the machined surface condition such as milling, grinding, chatter and remained tool mark, cluster analysis was adopted. The clustering results are evaluated by the clustering accuracy and statistical analysis, and these can easily evaluate the clustering result using a confidence ellipse [15]. For drawing the confidence ellipse, the Mahalanobis distance is defined as the distance between two variables and the covariance, as in Eq. (5). Covariance is defined as Eq. (6), which indicates the degree of correlation between two variables:

$$ {{\text{D}}}_{{{\text{M}}}} ({\text{X,Y}}) = [{({\text{X}} - {\text{Y}})^{\text{T}}} {{\text{C}}^{ - 1}} {({\text{X}} - {\text{Y}})]^{\frac{1}{2}}} $$
(5)

where D_M: Mahalanobis distance, X, Y: set of observations (mean and standard deviation), C: covariance matrix

$$ {\text{C(X, Y)}} = \left[ {\begin{array}{*{20}c} {\text{Var (X)}} & {\text{Cov (X,Y)}} \\ {\text{Cov (X,Y)}} & {\text{Var (Y)}}\\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} {{{\upsigma }}_{\text{x}}^{2} } & {{{\upsigma }}_{\text{xy}} } \\ {{{\upsigma }}_{\text{xy}} } & {{{\upsigma }}_{\text{y}}^{2} } \\ \end{array} } \right] $$
(6)

where C: covariance matrix, Var: variance of variable, σ: standard deviation of variable.

Also, the confidence ellipse can be defined as Eq. (7) using eigenvalues and eigenvectors of the covariance matrix and the standard deviation. The maximum and minimum eigenvalues represent the maximum and minimum radius of the confidence ellipse:

$$ \left( {\frac{\text{x}}{{{{\upsigma }}_{\text{x}} }}} \right)^{2} + \left( {\frac{\text{y}}{{{{\upsigma }}_{\text{y}} }}} \right)^{2} = {\text{S}} $$
(7)

where S: the value of the probability table (size of ellipse)

$$ \begin{aligned} 2\sigma_{x} \sqrt S \to 2\sqrt {S \cdot \lambda_{\rm{max} } } \hfill \\ 2\sigma_{y} \sqrt S \to 2\sqrt {S \cdot \lambda_{min} } \hfill \\ \end{aligned} $$

where λ: maximum and minimum eigenvalue of covariance matrix.

Figure 10 depicts the confidence ellipse using the Mahalanobis distance. The three different ellipse groups are independent of each other and also show whether the data in each group are trustworthy. Table 4 shows the covariance matrix of the machined surfaces.

Table 4 Covariance matrix of the machined surfaces
Full size table

The longest axis of the milling confidence ellipse represents the large standard deviation, because reflected light occurs in almost all areas and the influence on the orientation is larger than the magnitude of the intensity. On the other hand, all data exist in the confidence ellipse except one outlier among 80 data points using 8 images per surface.

In the case of plane grinding, the width of the ellipse is small because the range of the specular highlight is narrower than that of the other two machined surfaces. The hand grinder case becomes wider because the reflected light is split, and the surface has various intensities due to the non-uniformity. These results support that intensity-based clustering could be a good indication for distinguishing the machined surface state.

4.4 Detection of Tool Mark Using Specular Highlight

The use of the specular highlight is possible for determining the residual amount of tool mark for the surface inspection of each finishing step.

After converting the original image to a gray image, the reflected light image is removed as depicted in Fig. 11. Next, the tool marks and non-parts are separated through thresholding, leaving only tool marks. Since the boundaries and areas of tool marks are partial and ambiguous, expansion of the boundaries is necessary. Through the dilation technique, the edges of the tool mark are filled with blank parts to sharpen the boundary [17].

Fig. 11
figure 11

Image processing procedure for tool mark detection

Full size image

Next, the areas that are too small are removed for accurate tool mark detection and the boundary part of the remaining area is indicated by yellow.

Figure 12 shows that the tool mark around the reflected light is clearly visible, since the reflected light is the brightest part of the entire surface. By employing this technique, the residual tool marks around the reflected light can be detected by scanning the entire surface of image 19. But the scratch detection is unclear except for the tool marks when there is reflected light. Only scratches deeper than the depth of the residual tool marks can be detected.

Fig. 12
figure 12

Tool mark detection results of milled and lapped surfaces: a–c face milled respectively by 80 mm, 100 mm, and 125 mm, angle of light source 90°; d1–d3 lapped surface #600, angle of light source respectively 30°, 90°, 150°

Full size image

In the case of 125 mm (Fig. 12c), a wide range of tool marks is detected due to the chatter marks. In the case of the hand lapped surface with #600 (Fig. 12d1–d3), milling tool marks that have not been removed yet are detected. However, tool marks from lapping at the reflected light area are also detected. Therefore, the tool mark detection algorithm should be improved to distinguish between milling and lapping tool marks. Moreover, an algorithm capable of simultaneously detecting a scratch is also required.

5 Inspection with Photometric Stereo

5.1 Photometric Stereo

This technique was popularly employed in computer vision applications to estimate the surface normal of objects with observations under different lighting conditions [16]. It is formally defined by the surface normal from the reflected light between the object and the light source and the observed intensities, as given by Eq. (8):

$$ I = k(L \cdot n) $$
(8)

where I: vector of observed intensities, L: normalized light directions, n: surface normal, k: albedo reflectivity.

The intensity and the position of the light source, surface normal vector is derived from Eq. (8) through Eqs. (9) and (10). The surface normal is obtained through light modeling from images along the light source at different locations from Eq. (10):

$$ L^{T} I = L^{T} k(L \cdot n) $$
(9)
$$ (L^{T} L)^{ - 1} L^{T} I = kn $$
(10)

5.2 Surface Inspection Using Photometric Stereo

5.2.1 Setup for the Inspection

This setup is the same as the inspection configuration in Sect. 3.1 (Fig. 4), except for using two reflective balls and the light source position. This is because the reflective ball is an absolute reference for modeling the position of the moving light source. Iron balls were used in this study as the two reflective balls. Images are obtained at six locations of the optimized light source that do not cause specular highlight.

5.2.2 Optimization of Light Position for Removing Specular Highlight

The normal map is obtained as shown on the left of Fig. 13. This normal map does not recover the entire surface, and the left of Fig. 13 shows the surface information lost. That means that the photometric stereo is unsuitable for metal surfaces due to reflected light. If the normal map is acquired using the position of the light source where no reflected light is generated, a normal map that can recover the entire surface can be obtained, as shown on the right side of Fig. 13.

Fig. 13
figure 13

Normal map result according to presence (left) and absence (right) of reflected light

Full size image

Table 5 indicates the results of the horizontal angle region where reflected light occurs in Sect. 3.3 as well as the result of a region where no reflected light occurs. In this table, the common range of 13no reflected light is − 40° to 0° (40°) and − 140° to 180° (40°). As a result, six light source locations were set at 0°, − 20°, − 40°, − 140°, − 160°, and 180° as the common angle for six minimum images that can be recovered, using the opposite direction of Fig. 4b.

Table 5 The range where reflected light occurs or does not occur
Full size table

5.2.3 Normal Map Results of Machined Surfaces

As can be seen in Fig. 14, normal maps are acquired by inverting the z-direction of the original normal map and adopting it at 1.2 times the weight. The reason for using the z-direction normal map is that, since the machined surface is close to 2D, there is hardly any surface information in the x- and y-direction normal maps. The normal maps show the entire machined surfaces at once. But the middle region is relatively dark, because images near 90° and the opposite side are not considered in order to exclude specular highlight.

Fig. 14
figure 14

Normal map results: a1, a2 milled surfaces; respectively diameters of 100 mm, 125 mm; b1, b2 lapped surfaces; respectively grit size #600, #800; c1, c2 grinded surfaces; respectively plane grinder, hand grinder

Full size image

In the 125 mm milling case (Fig. 14a2), bright chatter tool marks are shown on both sides of the top, because there is faint reflected light even in the range where reflection should not occur due to chatter marks. For lapped with #600 (Fig. 14b1) or plane grinder ground surfaces (Fig. 14c1, c2), it looks noticeably better for deep scratches than for tool marks. In the case of hand lapping, there is the tool mark of the grinder itself and there are several highlights. This is also because the reflected light is generated in a range where it should not occur due to irregular tool marks by manual work.

5.3 Scratch Detection in the Machined Surface

In Sect. 4.4, since the tool mark around the reflected light was brightly captured, it was easy to find. However, it was difficult to detect scratches because of the bright characteristics of the reflected light. Therefore, adopting a normal map technique, it reflects the entire surface. The procedure is similar to that of the process in Sect. 4.4. As shown in Fig. 15 for the scratch detection, the inverted image was used in the original z-direction normal map.

Fig. 15
figure 15

Image processing procedure for scratch detection

Full size image

This facilitates banalization for leaving only scratches. Then, as in the process of Sect. 4.4, the edges of the scratch are expanded to sharpen the boundary through the dilation technique. Next, the areas that are too small are removed for accurate scratch detection, and the boundary parts of the remaining area are shown in blue.

In Fig. 16, the dark yellow boxes show where scratches are detected, and the yellow ellipses show where the milling tool mark, which could not be removed with #600 hand lapping (Fig. 16b), was detected.

Fig. 16
figure 16

Scratch detection results from normal maps: a milled surface, diameter 125 mm; b lapped surfaces, grit size #600; c grinded surfaces, plane grinder

Full size image

The lapped and ground surfaces may include unexpected scratches caused by handling. This scratch is rather deeper and clearer than tool marks such as the milling tool mark. To resolve this problem, a robust detection algorithm is needed to distinguish between scratches caused by external factors and the tool itself.

Significant improvements were made using the proposed method. The non-detection rate of scratches was very small and the improved inspection is superior to that from a human operator.

6 Conclusion

An image-based machined surface inspection based on an image processing technique was developed for applications in the process of unmanned lapping. In this paper, two methods are proposed. The first is an inspection method that utilizes reflected light characteristics by a tool mark on a machined surface. This distinguishes the machined surface state by using the shape of the reflected light and the intensity of the image according to the tool mark, and the tool mark can be also detected. The second is a normal map by the photometric stereo technique, which shows the entire surface at one time and can detect scratches much clear.

Consequently, if only the two methods are integrated, only six images can provide the classification of the machined surface state such as remained tool mark, waviness and scratch at once. This method can be applied to automate detection of the state of machined metal surfaces. This will widen the inspection method for robotic machining system as well as for unmanned machining processes.