Keywords

1 Introduction

The precise detection and analysis of visual anomalies are of primary importance in many fields involving the visual inspection of surfaces. This is particularly true in the case of metallic engineered surfaces where machining and finishing operations often produce some small and local defects on surfaces. Moreover, a solid trend in the industry of high added-value products consists in the automation of the visual inspection process. Indeed, various automated visual inspection systems for the detection of anomalies on metallic surfaces have been recently implemented [1], based on different set-up principles. In a non-exhaustive way, the dark field imaging technique consists of illuminating the surface by a directional light source at a grazing angle of incidence, whereas bright field imaging [2] uses a sensor placed along the direction of the specular reflection. As the curvature is known to correlate well with the visual perception, the D-Sight system [3, 4] focused on the detection of curvature variations by observing the reflected light through a retro-reflective screen. However, since the variety of potential visual anomalies is very large these systems are often designed for a specific kind of anomaly, finishing process, surface model, and/or object. Consequently, this leads to non detection of unexpected (out of the considered model) visual anomalies. Furthermore, these methods do not allow an accurate description of the visual anomalies, which are often compressed in a single descriptor (e.g. D-sight curvature maps), or reduced to an intensity-based variation in the used imaging modality.

We deal with this issue by proposing a methodology based on the detection and the analysis of the changes on the local “visual behaviour” of the surface using reflectance information. The detection is indeed eased by the utilization of reflectance features on the inspected surfaces, which are very discriminating when calculated locally (on each point of the surface) and made invariant to rotation. To achieve the acquisition of reflectance characteristics, the method proposed in this work is placed within the framework of the techniques called Reflectance Transformation Imaging (RTI) which are widespread in the field of Cultural Heritage for the digitalization of the visual appearance of objects [5]. In the RTI technique, for each pixel of the inspected surface, a set of values representing the reflected light for each light direction is acquired allowing the modelling of what we call the angular reflectance (in contrast with spectral reflectance). The approximation of this discrete set of values by a surface is a least square regression for which we propose the method based on the Discrete Modal Decomposition (DMD) [6]. This latter provides indeed a fine description of the local angular reflectance particularly in the case of local sharp variations of angular reflectance, such as specularities, shadows, or sparkles, which can induce visual anomalies by disrupting the reflectance spatial homogeneity. We then introduce a new rotation-invariant representation derived from the DMD that enables the method to be insensitive to the object/sensor relative orientation during acquisition, or to the orientation of the intrinsic surface roughness pattern. Saliency maps are then estimated from this information as means of defects detection and characterisation. Experimental results are presented on different engineering metallic samples exhibiting several types of appearance defects and compared to other conventional saliency assessments.

2 Related Work

The most exhaustive description of the reflectance is the Bidirectional Reflectance Distribution Function [7] (BRDF). However, the assessment of this function is costly in terms of processing time and amount of data, and is therefore not adequate for the detection of visual anomalies. A simplification of the BRDF consists in the acquisition of the angular component of the reflectance– the sensor is in a fixed position normal to the surface while only the spatial positions of lighting vary during acquisitions – leads to achieve a good balance between a fine description of local visual behaviour and the practicality. We propose thus to implement RTI technique along with its subsequent processing for aiding the visual inspection of surfaces by allowing the detection and description of visual appearance anomalies in terms of visual behaviour.

2.1 RTI Data Acquisition Requirements

As the high quality of the source images is essential, we first briefly recall the main requirements for RTI stereo-photometric data acquisitions, before detailing the next steps of our methodology. An RTI setup is used to acquire for each pixel of the scene the luminance values \(\mathcal {L}\) under varying light positions defined in the coordinate system \((l_u, l_v,\mathcal {L})\), where \((l_u, l_v)\) are the components associated with light positions (LP) projected in the horizontal plane, as shown in Fig. 1 (left). The sensor is generally fixed at the centre of a dome [8], orthogonal to the inspected surface. Other RTI setups based on this principle have been designed to adapt the capture of image sources from in situ archaeological material to the intrinsic constraints of the object and its environment [9, 10].

Fig. 1.
figure 1

RTI principle. (left) Angular reflectance approximated from the measured luminances expressed in the coordinate system \((l_u, l_v,\mathcal {L})\) and (right) our RTI setup

Full size image

2.2 Angular Reflectance Modelling

Modelling of the angular reflectance from RTI acquisitions aims at approximating the set of the discrete points representing the luminance values at each pixel for all light positions (hemispheric shaped) by a continuous surface as shown in Fig. 2 (right). This step is called surface parametrization for which, to our knowledge, only three methods exist in literature: PTM [8, 11], HSH [12] and DMD [6]. The objective of the parametrization is to obtain the best fitting of the information held by the discrete points while compacting it reliably. We chose to implement here the Discrete Modal Decomposition (DMD), which outperforms the others, especially in detecting and highlighting singularities on surfaces [6]. The DMD method is based on a projection on the Modal bases. The modal bases are composed of elementary forms that take their origin of a structural dynamic problem. We adapt these modal bases to the case of angular reflectance reconstruction by devising new modal shapes that we call Reflectance Modal Basis (RMB), solution of the structural dynamic problem whose reference surface is a hemisphere. This set of elementary forms noted \(Q_i\) (with \(i = 1..m\) where m is the number of modes) is used to reconstruct the reflectance shape from discrete values (luminance) obtained during the acquisition stage [13]. A representation of first modes of the RMB is given in Fig. 3. Therefore, the local reflectance associated to each point/pixel of the inspected surface is then described by a modal spectrum representing the contributions \(\lambda _i\) of each mode of the RMB (Fig. 2 - left).

Fig. 2.
figure 2

Angular reflectance modelling at a pixel (right) approximated from measured luminances (blue dots) using DMD method and (left) the associated spectrum \(\lambda _i\) representing the contributions of each mode of the RMB (Color figure online)

Full size image

3 Proposed Methodology

In the following we show how we extend DMD in order to automate visual inspection by introducing a rotation-invariant representation of the reflectance characteristics of any inspected surface.

3.1 Rotation-Invariant Representation

One interesting property of the modal shapes is that it can be separated in rotation-dependent and rotation invariant modes, named respectively simple and congruent modes. The shape of the simple modes presents a rotational symmetry which preserve their shape for any rotation around the vertical axis (modes \(Q_1\) or \(Q_{6}\) in Fig. 3). They are thereby originally rotation-invariants. Some pairs, namely the congruent modes, show the same shape but are oriented differently (for instance \(Q_{2-3}\) or \(Q_{4-5}\) in Fig. 3). As a consequence, in this original form, the modal spectrum composed of \(\lambda _i\) coefficients will then vary when the orientation (rotation for example) of the object under the camera is modified. The same principle applies to, for example, a sample with a regular pattern oriented differently across its surface: two points with the same visual appearance behaviour but different orientation will be described by completely different modal spectra. In order to use the modal information for the estimation of visual saliency maps, it is thus essential to transform the original modal shapes to make them rotation-invariant.

Fig. 3.
figure 3

First descriptors of the RMB noted \(Q_{i}\), with the associated phase-angle \(\alpha _i\) for the congruent modes

Full size image

The proposed method to obtain a rotation-invariant representation is based on the separation of the phase and amplitude components. This separation is performed through a change applied on the congruent modes group presented below. The amplitude and the phase-angle (resp. \(\lambda _j'\) et \(\alpha _j'\)) can be derived from the expression of the linear combination of two congruent modes \(Q_i\) and \(Q_{i+1}\) and their modal coefficients \(\lambda _i\) and \(\lambda _{i+1}\) as shown in the Fig. 4. The resulting amplitude \(\lambda _j'\) is obtained by computing the L2-norm: \(\lambda _j'= (\lambda _i^2+\lambda _{i+1}^2)^{1/2}\).

Fig. 4.
figure 4

Change of parametrization in the amplitude/phase-angle form (e.g. \(Q_{2-3}\) pair)

Full size image

The L2-norm does not vary when the orientation of the reflectance is rotated around the vertical axis of \((l_u,l_v,\mathcal {L})\). The vector of resulting amplitudes \(\lambda _j'\) is therefore a subset of the initial coefficients \(\lambda _i\) which characterizes the shape of any reflectances independently of its spatial orientation. In other words, the angular reflectances which are identical in shape but differently oriented, i.e. which can be brought into coincidence by rotation, present the same modal spectra in this rotation-invariant reflectance description. The phase-angle \(\alpha _j'\) of the associated \(Q_j'\) mode can be determined by the following Eq. 1, where the phase-angle \(\alpha _i\) of each mode is known a priori and saved with the RMB.

$$\begin{aligned} \alpha _j' = \alpha _i + \arctan { \frac{\lambda _{i+1} sin{(\alpha _{i+1}-\alpha _i)} }{ \lambda _i + \lambda _{i+1} \cos {(\alpha _{i+1}-\alpha _i)} } } \end{aligned}$$
(1)

This phase information is not used for this methodology, but could be relevant for other applications, as for example to assess the orientation of a textural pattern, or to determine the appropriate light directions in order to find specific visual appearance features on surfaces.

3.2 Visual Saliency Estimation

In order to enable the identification of atypical (salient) behaviours in term of angular reflectance, we implement a multivariate statistical analysis on the local invariant modal spectra. The multivariate image is defined as a three-dimensional \(n_1 \times n_2 \times m_\mathcal {B}\) data matrix, where two spatial dimensions represent the \(n_1 \times n_2\) image pixels and the third dimension represents the \(m_\mathcal {B}\) modal rotation-invariant coefficients \(\lambda _j'\). We can reshape the \(n_1 \times n_2-\)pixel images of the amplitude series into column vectors with dimension \(n=n_1n_2\) and express the multivariate image data as two-dimensional \(n\times m_\mathcal {B}\) matrix \(\varvec{\lambda '} = [\lambda _1',\lambda _2',...,\lambda _{m_\mathcal {B}}']\). Then, we use the Mahalanobis distance \(D_{\text {Mahal}}\) [14] to assess the distance between the reflectance modal spectrum of each element (pixel) and the average modal spectrum of reflectance estimated on the whole image, i.e. from all vectors \(\varvec{\lambda '}\). The expression of \(D_{\text {Mahal}}\) is given by:

$$\begin{aligned} D_{\text {Mahal}} = \sqrt{(\varvec{\lambda '}-\varvec{\mu })^T\varvec{\varSigma ^{-1}}(\varvec{\lambda '}-\varvec{\mu })} \end{aligned}$$
(2)

where \(\varvec{\varSigma }\) and \(\varvec{\mu } =(\mu _1,\mu _2,\mu _3,...,\mu _p)^T\) are respectively the covariance matrix and the average vector describing the mean reflectance shape across the surface. The visual saliency is finally computed from this distance estimation on each point of the inspected surface and plotted as saliency maps. Figure 5 summarizes this methodology.

Fig. 5.
figure 5

Methodology flow chart

Full size image

4 Results and Discussion

In order to evaluate the proposed method, we tested it on different metallic objects (samples). We compared our results with both human observers assessment and a state-of-the art method for saliency maps computation. Since the state-of-the-art method was not initially adapted to RTI data and objects we studied, we introduced an adaptation on it. Then the rotational invariance of our methodology is demonstrated on a real case. The samples described in Sect. 4.1 were acquired with a dome with 96 light sources evenly distributed over the hemisphere presented in Fig. 1 (right).

Table 1. Geometric parameters of the indented scratch and results of the sensory analysis on \(\mathcal {S}_{1-3}\) samples [15]
Full size table

4.1 Datasets

 

Dataset \(D_1\) . :

The first dataset we used presents the interest that the scratchs introduced on it are controlled (well known features and locations). The samples are polished metallic surfaces (Fig. 6 on top) which were indented with a Berkovish nanoindenter. The indented scratches of the three tested samples are described by the geometric parameters presented in Table 1. These samples were assessed through a sensorial analysis performed by trained quality controllers, as described in [15]. The adaptation phenomenon [16] during this sensory evaluation was notably reduced by positioning the scratches at random locations on each sample (Fig. 6). A visibility ratio is thereby associated to each sample (Table 1). As other uncontrolled degradations are observed on the samples, these defects present a good challenge for our approach.

Dataset \(D_2\) . :

The second dataset corresponds to a gauge block that is used as a standard reference to check the calibration of measurement tools such as micrometers. The manufacture of gauge blocks implies lapping and polishing processes with high precision for achieving a fine surface finish. These gauges have to be of very high quality (precision and location of patterns) which requires a strong quality control (anomaly detection) stage. One side of the gauge block of the dataset \(D_2\) has an unidirectional brushed pattern. For an identical scene of observation (light position and view directions) given in Fig. 7 a, the surface appearance of the object changes when its orientation (rotation) varies. The rotation of the object is thus not equivalent to a rotation of the image. However, the angular reflectance associated to a point of the object in any spatial orientation has to be identical, only phase shifted.

 

Fig. 6.
figure 6

Saliency maps \(\mathcal {S}_{1-3}\) obtained, from RTI data, with (a) the Itti and Koch’s saliency model and (b) our method

Full size image

4.2 Anomaly Detection Performances

The resulting saliency maps obtained with our method (Sect. 3) are given in Fig. 6 b for each sample of the first dataset \(D_1\). Beside, in a purpose of comparison, we use a well-known implementation of Itti and Koch’s saliency model (Matlab Saliency Toolbox [17, 18]) based on three visual features - luminance contrast, orientation and colour - with each feature computed over a multi-scale Gaussian pyramid. Itti’s model was largely extended to the saliency estimation in a dynamic scene (video) by integrating motion estimators [19], applications on 3D meshes [20], or multispectral content [21]. As this model was optimized for a single image scene with macroscopic objects, a change is thus applied on the parameters related to the scale levels (\(c \in \left\{ 1, 2, 3, 4 \right\} \) and \(\delta \in \left\{ 1, 2, 3 \right\} \)). These parameters were validated on synthetic rough surfaces [22, 23] in order to be sensitive to high frequency spatial information contained in the surface images. The Itti’s method is then applied on each input image of the RTI dataset in order to generate a saliency map associated with the different light directions, and then all of those maps are combined in a final saliency map (Fig. 6 a). An effective separation of the scratches on the polished region correlated with the visibility (Table 1) is obtained by the two approaches. While the \(\mathcal {S}_3\) calibrated scratch is quite invisible for human inspection, our method succeeds to reveal its proper signature (straight line). The results on these samples demonstrate the effectiveness of our saliency maps as means of anomaly detection. With regards to Itti’s methods, the saliency maps generated are rather rotation-dependent since the small scratches are not detected. In addition, major limitations of Itti’s method are related to its computational complexity and time costs of processing all images of RTI data structure. In contrast, our method allows significant time savings (150 times faster) thereby satisfying time requirements in many inspection applications of real-world surfaces.

Fig. 7.
figure 7

Effects of the sample orientation change. (a) The same sample acquired for the same light position for three orientations (0, 60 and 90\(^\circ \) rotation): (b) the corresponding estimated angular reflectance, (c) the saliency maps using our methodology

Full size image

4.3 Invariance to Object Rotation

Our approach is applied on the second dataset \(D_2\) for illustration purposes of the rotation invariance property of our methodology. This rotation-invariance property is important for the comparison between the modal spectra corresponding to all point/pixel of the inspected surface. Those spectra describe the reflectance shapes, and therefore allow to generate robust saliency maps that are not affected by the object pose or the orientation of texture pattern. The mean reflectance shape (Fig. 7 b) characterizing the normal behaviour over the whole surface of the gauge block corresponds to the linear combination of the mean vector of coefficients \(\lambda _i\) with the associated modes \(Q_i\) of the RMB. In this original space of representation, we can easily observe that the spectra \(\lambda _i\) vary while the surface orientation of the gauge block is rotated, as shown in the first chart of Fig. 8. The change of parametrization consists in replacing the set of coefficients \(\lambda _i\) by a subset of amplitudes \(\lambda _j'\) (second chart of Fig. 8), after separating the component linked to the orientation as described in Sect. 3.1. The resulting spectra of amplitudes \(\lambda _j'\) constructed from the initial set of coefficients \(\lambda _i\), are quite similar (invariant) for the three different orientations (0, 60 and 90\(^{\circ }\) rotation) of the gauge block. The small differences between these spectra can be explained by the light positions that can not be strictly identical in practice when the surface is rotated inside the RTI dome. However these deviations are not critical for the visual saliency estimation. The resulting saliency maps given in Fig. 7 c obtained from these data show that our approach is robust to a change of orientation of the object. The salient features detected over the metal surface of the gauge block corresponds to many scratches, marks, and regions where the brushed pattern displays more pronounced (deeper) lines than normal visual appearance.

Fig. 8.
figure 8

Change of parametrization from the original spectra \(\lambda _i\) to its rotation-invariant spectra \(\lambda _j'\), characterizing the reflectance shape independently of the surface orientation

Full size image

5 Conclusion

This paper presented a novel methodology for the detection of anomalies applied on metallic surfaces from high dimensional RTI data. The method presents the advantage to be insensitive to both object rotations and pattern orientation. The effectiveness of our methodology in terms of both detection performance and of time-saving was shown on real surfaces. We claim:

  1. 1.

    A change in the DMD parametrization of angular reflectances was introduced in order to compare the reflectance shapes independently from their spatial orientation.

  2. 2.

    Changes in reflectance shape over the inspected surface are then identified and located by performing a multivariate analysis in this rotation-invariant space.

  3. 3.

    Finally, we provided a method for saliency maps computation from this information as means of anomaly detection. The maps we compute showed a precise location of surface anomalies. Plotting reflectance shapes at the location of detected anomalies could be used in future work as a descriptor correlated with the anomalies visibility for the criticality evaluation.