Keywords

1 Introduction

Quality control for manufactured parts in the automotive, aeronautics, and energy sectors is performed by certified metrological laboratories, often by using 3D laser scanners. The part under study is captured in a set of points constituting a point cloud that is the virtual representation of the actual object. The point cloud is filtered to reduce noise and then processed to achieve the final goal of the study, which is dimensional and/or positional measurement and tolerancing. Based on the geometry of the features of the object, the metrologist initially constructs a measurement plan. However, this preliminary study may need to be repeated for less-experienced operators who are less likely to define an appropriate measurement plan from the beginning. This obstacle may lead to both time-consuming processes and inconsistent results between operators.

The goal of this study is to model the differential accuracy of the laser scanning device in response to different geometries and under various scanning configurations. Capture errors are driven by several sources, such as the incident angle, measurement distance, and surface texture. Measurement error is introduced by the instrument itself, which has an anisotropic response by design: accuracy is maximum in the axial direction (axis of movement of the rotary head) and diminishes laterally. Another important source of errors can be the surface orientations that the object’s geometry encompasses with respect to the laser source orientation and the viewing direction of the CMOS sensor. In essence, very small incidence angles of the laser light on the surface (i.e., when surface and laser source orientations are quasi-parallel) result in severe backscattering of the light toward the CMOS camera, so large capture errors are introduced. On the other hand, very large incidence angles (close to 90°) also introduce large errors, but for a different reason: most of the laser light is scattered away from the CMOS sensor rather than being collected to capture an accurate representation of the surface.

To address these issues, we propose a novel approach that leverages the power of artificial intelligence (AI) and a specific subfield known as Explainable AI (XAI). XAI is an emerging area of AI that aims at making the decision-making process of AI models transparent and understandable. Unlike traditional AI, where decision-making processes can be black boxes, XAI allows users to understand, trust, and manage AI outcomes. By breaking down complex AI decisions into digestible, human-interpretable components, XAI tools help enhance trust and improve adoption of AI systems. In this context, we propose a novel approach to assist metrologists in defining the optimal scanning setup for the measurement. This approach focuses on the geometrical properties of the objects and enables manufacturers to save time and costs while providing consistent results. We develop an AI-based decision support system to predict the point-wise accuracy of the laser scanner across the surface of the part under analysis. We further apply an explainability tool to provide a comprehensive view of the most important parameters affecting the model’s predictions. These user-centered explanations are the key ingredients in building trust and allowing inexperienced operators to have better understanding about the different responses of the scanning device and make informed decisions.

The rest of this chapter is organized as follows: the necessary background on optical industrial metrology as well as XAI applications in industry 4.0 is provided in Sect. 2; the formulation of the problem and the available data sources are outlined in Sect. 3; The proposed methodology and experimental setting are explained in detail in Sect. 4; experimental results and associated explanations are presented in Sect. 5; and conclusions and practical implications are presented in Sect. 6.

2 Background and Related Work

2.1 Optical Metrology

Optical measurement in digital manufacturing has been extensively researched, with a growing emphasis on integrating metrology into the production flow to optimize processes and enable fully automated manufacturing cells [1]. Laser-based instruments have emerged as the most common technology for industrial applications, although on-machine inspection still faces challenges due to high processing temperatures and random process variations. Key trends in this field include the shift toward zero-defect manufacturing strategies, facilitated by in-line measurement and in-process monitoring. However, several challenges must be addressed to ensure effective and efficient integrated metrology, including limitations in measurement and data processing speed, part complexity (size, shape, and surface texture), user-dependent constraints, and measurements in challenging environments [2].

Measurement speed is crucial, as faster measurements without sacrificing accuracy and precision lead to increased throughput and reduced production costs [3]. Addressing part complexity necessitates adaptable and versatile measurement solutions to accommodate various manufacturing scenarios [4]. Challenging environments present additional difficulties, requiring alternative design strategies and robust sensors capable of reliable in situ operation [5]. Furthermore, multi-sensor integration and data fusion are essential to enhance overall measurement quality and provide comprehensive information about parts and processes [6]. This demands robust and efficient methods for sensor compatibility, data synchronization, and data fusion techniques. Addressing these challenges is vital for advancing integrated metrology and improving its effectiveness in modern manufacturing.

Optical inspection methods, particularly laser triangulation measurements, have developed significantly in recent years, providing fast data acquisition, contactless measurements, and high sampling rates. However, challenges remain with the accuracy and reliability of these measurements. The random measurement error is an important aspect to consider, as it quantifies the variation of the actual measurement from its expected value [7]. Factors that affect the measurement error of scanners include the angle of inclination [8], sensor distance from the surface, color [9], texture, and surface reflectiveness [10]. Various studies have investigated the impact of different factors on measurement performance and proposed methods to improve the quality of this process. These include the use of least-squares methods [11], color-error compensation methods [12], and mathematical frameworks for statistical modeling of uncertainties [13].

Some machine learning (ML) methods have also been applied to determine the measurement capacity of scanning devices and improve processes related to non-contact 3D scanning [14,15,16]. These ML applications include simplifying information about geometry obtained from point clouds and identifying distinct geometrical features of objects. With the ongoing advancement of optical inspection methods, researchers are investigating various techniques to improve the accuracy and reliability of measurements across various contexts [17]. Despite the numerous advantages and improvements offered by ML methods, there remains a notable gap in the current literature. A major concern is that most ML models are considered “black boxes,” meaning their inner workings and decision-making processes are not easily interpretable or transparent. Transparency is of crucial importance in various industries, including metrology, since it ensures that stakeholders can trust the results generated by these models. When implementing ML techniques, it is vital to have a clear understanding of how the models arrive at their conclusions, particularly when making critical decisions that directly impact product quality, safety, or compliance with industry regulations. The lack of transparency in current ML models raises questions about their reliability and suitability for widespread adoption in the field of metrology.

To address this gap, our work focuses on developing more transparent and interpretable ML models that can be easily understood by practitioners and stakeholders in the metrology industry. By providing transparency for ML models, researchers can foster greater confidence in their adoption, leading to more accurate and reliable measurements and, ultimately, enhancing the overall quality of optical inspection methods.

2.2 Explainable Artificial Intelligence

The research and application field of XAI has been very active, following the widespread use of AI systems in many sectors related to everyday life and the rising need for interpreting the predictions of these complex systems. Interpretable by design models such as Decision Trees and linear models are outperformed by more complex methods such as Neural Networks, Support Vector Machines (SVMs), or ensemble models. The term XAI mostly refers to post-hoc explainability being applied to interpret-trained ML/DL models with complex internal mechanics that are otherwise opaque. Many different approaches have been proposed to this end (see for example [18] for a comprehensive review), and although taxonomies in the literature are not identical, a first-level distinction of XAI methods mainly pertains to the following:

The applicability of the method: Model-agnostic methods are methods applicable to any kind of model, whereas model-specific methods are designed to explain a certain type of model, considering their particular properties and internal design.

The scale of explanations: global methods explain the overall behavior of the model (at the global level), whereas local explanations refer to individual predictions (at the instance level).

XAI methods are further classified by their scope, with those related to our study being feature attribution methods that aim to quantify the effect of individual input features on the model’s output. Global techniques estimate feature importance by the overall change in model’s performance in the absence of an input feature or under shuffling of its values, such as permutation importance [19]. Local feature attributions, on the other hand, explain how individual predictions change with each input feature. Shapley Additive eXplanations, or SHAP [20], apply methods from cooperative game theory to machine learning. Like Shapley values measure a player's impact in a game, SHAP values measure each input feature's impact on a model's prediction, compared to the average prediction. Local explanations are then aggregated to provide the ranking of global feature importances.

SHAP is a well-established, model-agnostic method with robust theoretical background that has been widely employed to interpret the predictions of ML models across various application domains, including the industrial sector. In the field of machine prognosis and health monitoring, the authors of [21] designed a deep-stacked convolutional bidirectional Long Short-Term Memory (Bi-LSTM) network. This network predicts a turbofan engine's Remaining Useful Life (RUL) using time-series data measured from 21 sensors. They apply SHAP values to identify the sensors that are mostly affecting the prediction and produce rich visualizations to meticulously study the explanations. In a similar line of work, [22] used SVM and k-Nearest Neighbor models on vibration signals to achieve fault diagnosis of industrial bearings. They depict the most important features influencing fault prediction with SHAP. An XAI approach for process quality optimization is proposed by [23]. They train a gradient-boosting tree-based ensemble on transistor chip production data of Hitachi ABB and identify the most significant production parameters affecting the industry’s yield using SHAP. A detailed study on how these parameters affect the production allowed for the prioritization of processes and the selection of improvement actions. Experimental validation of the method on a new production batch showed a remarkable improvement in yield losses, leading the company to adopt the method for further use. These indicative studies show the added value brought about by the SHAP method explaining complex models’ predictions, allowing thus otherwise “black-box” models to be transparent to stakeholders.

Despite the active research and applications of XAI in various sectors, its application within the realm of metrology remains relatively unexplored. Specifically, there is a notable gap in the literature regarding work focusing on developing an explainable pipeline for predicting point-wise accuracy of laser scanning devices in the metrology domain. This chapter aims to bridge this gap. Our work contributes to the nascent field of “Explainable Metrology 4.0” by designing a robust, interpretable model that not only predicts the accuracy of laser scanning devices but also provides valuable insights into the features that drive these predictions. By incorporating XAI techniques, particularly the model-agnostic SHAP, into metrology, we aim to transform “black-box” models into transparent tools that empower stakeholders to understand and trust the AI-driven measurement process. This novel approach is envisioned to bring about a significant positive shift in the field of industrial metrology.

3 Methodology

The proposed methodology is described step by step in this section, from data preparation and feature extraction to the established pipelines for model training, optimization, evaluation, and explanation.

3.1 Setting Up a Supervised Learning Task

Our study aims to model the behavior of the scanning instrument, taking into account both the scanning configuration and surface orientation. Specifically, our objective is to predict point-wise measurement errors at individual points along the three axes (X, Y, and Z). This prediction is a function of the scanning conditions, surface orientation, laser orientation, and the viewing direction of the CMOS sensor. Therefore, we can formulate the problem as:

$$ \textbf{measurment}\ \textbf{error}=\boldsymbol{f}\left(\begin{array}{c}\textbf{scanning}\ \textbf{conditions},\textbf{surface}\ \textbf{orientation},\\ {}\textbf{laser}\ \textbf{orientation},\textbf{CMOS}\ \textbf{viewing}\ \textbf{direction}\end{array}\right) $$

Having defined the problem to be solved, we seek to establish a supervised learning setting to address its solution. Supervised learning requires access to a certain ground truth that the models are trained to predict. This is achieved with the use of calibrated object measurements. Calibrated artifacts have properties known to be a very high level of precision (of the order of 1 μm = 10−6 m). Characteristic properties include dimension/position of features, while shape perfection is also certified. For these objects, we can safely consider that the actual properties, such as dimension, are identical to the nominal ones (provided by the manufacturer). Assuming that the center of the calibrated object is perfectly positioned at the center of the scanning instrument’s reference frame, the problem can be formulated as a supervised regression task, where the ground truth is found in the location of points (in Χ, Υ, Ζ coordinates) on the nominal surface of the calibrated object.

3.2 Data Sources

Our database so far comprises measurements of three calibrated spheres (three diameters), under different scanning configurations.

Scanning configurations are obtained by variation between three levels (low–medium–high) of the following scanning conditions:

  • Lateral density: Point density in the lateral direction with respect to the movement axis of the rotary head.

  • Direction density: the velocity of the rotary head, inversely proportional to point density along the axial direction (axis of movement).

  • Exposure time: the duration of each laser pulse.

As a result, we have an initial database comprising 108 raw point cloud files (3 objects × 33 scanning configurations). The number of points in each point cloud varies (min: 16 k, max:180 k) depending on the size of the object and scanning conditions. The total number of points adds up to ~6.5 M.

3.3 Data Pre-processing

Preliminary processing of raw Point Clouds is performed with the Open3D open-source library [24]. Open3D offers a rich collection of data structures and geometry processing algorithms to support the analysis of 3D data. With the use of Open3D functions, two key processing steps are performed:

  • Statistical outlier removal: Points that are, on average, further away from their neighbors are described as outliers. Configurable parameters are the number of nearest neighbors and the standard deviation ratio to be considered in the calculations.

  • Estimation of surface normal: The normal vectors are estimated for points in the point cloud based on the number of nearest neighbors within a given radius (these are configurable parameters). Once normal vectors are calculated, a second function is applied to ensure that they are consistently aligned. The alignment of surface normals is verified via visual inspection.

3.4 Feature Extraction

Based on the estimated normal vectors that stand for the surface orientation at each point, a set of informative features is extracted to describe the geometric setup between (a) surface orientation \( \overrightarrow{N} \), (b) the laser source orientation \( \overrightarrow{L} \), and (c) the CMOS sensor viewing direction \( \overrightarrow{V} \). Geometric features include cosine similarities between vectors and vector differences. In addition, the vertical distance between each point and the laser source is calculated, as well as the four-quadrant angle between the laser and the surface orientation (Fig. 1).

Fig. 1
Top. An illustration of an optiscan with laser orientation and viewing direction. Bottom. A vector diagram of the C MOS has a circle with the coordinate C of 0, 0, 0. A diagonal line for the N vector is drawn from the origin, and the points P and P prime are marked on the line.

The vectors contributing to the measurement geometric setup are surface orientation (N), laser orientation (L), and CMOS viewing direction (V). Each measured point P is projected along surface orientation to retrieve the closest point P′ on the nominal surface

Full size image

Overall, the input features to the ML algorithms are listed below:

  • Scanning conditions: lateral density, direction density, and exposure time

  • Components of surface normal:

$$ \overrightarrow{N}=\left[{N}_x,{N}_y,{N}_z\right]\vspace*{-10pt} $$
(1)

  • Components of the difference vector \( \overrightarrow{ori} \):

$$ \overrightarrow{ori}=\overrightarrow{N}-\overrightarrow{L}=\left[{ori}_X,{ori}_Y,{ori}_Z\right] \vspace*{-8pt}$$
(2)

  • Components of the difference vector \( \overrightarrow{oriCMOS} \) (we use only the Y-component, because X & Z components are equal to those of \( \overrightarrow{ori} \) above by design):

$$ \overrightarrow{oriCMOS}=\overrightarrow{N}-\overrightarrow{V}=\left[{oriCMOS}_X,{oriCMOS}_{\textrm{Y}},{oriCMOS}_Z\right] $$
(3)

  • Cosine similarity Inc between \( \left(\overrightarrow{N},\overrightarrow{\textrm{L}}\right) \), that is, the cosine of the incidence angle of light on the surface:

$$ \textrm{Inc}=\cos \left(\frac{\overrightarrow{N}\bullet \overrightarrow{L}}{\left\Vert \overrightarrow{N}\right\Vert \left\Vert \overrightarrow{L}\right\Vert}\right) $$
(4)

  • Cosine similarity ViewAng between \( \left(\overrightarrow{N},\overrightarrow{\textrm{V}}\right) \), that is, the cosine of the viewing angle of the surface from the CMOS camera:

$$ ViewAng=\cos \left(\frac{\overrightarrow{N}\bullet \overrightarrow{V}}{\left\Vert \overrightarrow{N}\right\Vert \left\Vert \overrightarrow{V}\right\Vert}\right) $$
(5)

  • The vertical distance Rs from each point to the laser source, which is always at Zlaser ≈ 90 mm:

$$ {R}_s=90-Z $$
(6)

  • The four-quadrant angle ang between laser orientation and surface orientation:

$$ ang=\mathit{\arctan}2d\left(\left\Vert \overrightarrow{N}\times \overrightarrow{L}\right\Vert, \overrightarrow{N}\bullet \overrightarrow{L}\right) $$
(7)

The target variables, i.e., point capture errors in the x, y, z axes, are defined by projecting captured points on the nominal surface of the calibrated object. The projection is radial in the case of the sphere and axial for the cylinder (the cylinder’s axis lies along the Y-axis of the instrument’s reference frame). In practice, by projecting each measured point P(X, Y, Z), we find the closest point P′(Xnom, Ynom, Znom) on the nominal surface of the calibrated object. For this “reference” point we can assume that:

  • If it belongs to a sphere, its actual radial distance to the center of the sphere C(0, 0, 0) is equal to the nominal radius of the sphere Rnom. This can be compared to the radial distance R of the measured point P, where:

$$ R=\sqrt{X^2+{Y}^2+{Z}^2} $$
(8)

  • If it belongs to the cylinder, its axial distance (distance to the Y-axis) is equal to the nominal radius of the cylinder, Rnom. This can be compared to the axial distance R of the measured point P, where:

$$ R=\sqrt{X^2+{Z}^2} $$
(9)

This analysis is based on the underlying assumption that the center of the sphere is perfectly positioned at the center of the scanning instrument’s reference frame and that the cylinder’s axis is perfectly aligned to the Y-axis of the reference frame.

The coordinates of the reference/projected point P′ for the sphere are calculated as follows:

$$ {X}_{\textrm{nom}}=X\frac{R_{\textrm{nom}}}{R} $$
(10)
$$ {Y}_{\textrm{nom}}=Y\frac{R_{\textrm{nom}}}{R} $$
(11)
$$ {Z}_{\textrm{nom}}=Z\frac{R_{\textrm{nom}}}{R} $$
(12)

The same formulas as above also apply to Xnom and Znom for the cylinder, whereas Ynom = Y, since the projection is axial in this case.

It follows that the target variables for our models, that is, point-wise measurement errors in three axes, can be computed as the difference between projected (nominal) and measured point coordinates:

$$ Xerr={X}_{\textrm{nom}}-X $$
(13)
$$ Yerr={Y}_{\textrm{nom}}-Y $$
(14)
$$ Zerr={Z}_{\textrm{nom}}-Z $$
(15)

Obviously, Ydev is applicable to spherical geometry, while for the cylinder Ydev = 0.

Equations 13, 14 and 15 imply that, in the positive X, Y, Z coordinate range, negative errors correspond to points captured outward to the surface, while points with positive errors are captured inward to the surface. The situation is reversed in the negative coordinate range.

Models will be trained and optimized to predict capture errors in each axis, with the same input features, while the target variable will be the measurement error in the respective axis.

3.5 Model Training, Optimization, and Validation

The original PointNet architecture [25] was proposed to deal with tasks pertaining to the segmentation and classification of point clouds. Taking n points as input (each point represented by its spatial coordinates), it initially feeds the input points in a Spatial Transformer Network (STN), with the aim of endowing the overall model with invariability with respect to affine transformations, specifically rotations and translations. The learned features are then passed through a Max Pooling layer, producing a global feature vector. In classification-oriented tasks, this global feature vector is fed to a standard Multi-layer Perceptron (MLP) which outputs scores for each one of the classes under consideration; in segmentation-oriented tasks, an additional step takes place, whereby the global features produced by the Max Pooling layer are concatenated with the output of the last layer of the model, resulting in a feature vector comprising both local and global features. A final pass through a second MLP takes place, and the model outputs a label for each point, indicating the object it belongs to.

In order to utilize PointNet for the prediction of deviations for each point, certain modifications must first take place:

  1. 1.

    Since the problem we are dealing with is neither classification nor segmentation, but regression, the output layer of PointNet must be modified accordingly to reflect the difference in objective. Therefore, the final layer of the model is replaced by a Linear Layer, and the evaluation metric which drives the updates of the model’s parameters is changed to the MAE function. The model outputs the deviation of each point in a specific axis; therefore, three distinct models must be trained in order to obtain the overall deviation in 3D space.

  2. 2.

    The employment of the Spatial Transformer Network is redundant in our case, since spherical objects are by default invariant to rotations, and the objects in consideration are not subject to potential translations. Having to learn additional parameters for the STN submodule would only lead to overfitting of the overall model. Therefore, the first step of passing the inputs through an STN submodule is skipped (Fig. 2).

Fig. 2
A block diagram exhibits the framework of the PointNet model. It consists of input points, M L P, max pooling, global feature, linear layer, and point deviation.

Modified PointNet Architecture. The network takes n points as input, passes the data through two series of MLPs, and extracts a global feature vector through a MaxPooling layer. The global feature vector is concatenated with the output of the first MLP sub-module and a final MLP is used to perform regression on point deviations

Full size image

In training the model, grid search and cross validation were performed for the final MLP submodule. Specifically, a range of different numbers of layers and the respective units comprising each layer were tested to identify the optimal combination. The Rectified Linear Unit (ReLU) activation function is used for all MLP layers, and the learning rate is set to adaptive in order to avoid training stagnation.

3.6 Comparative Analysis

An extended search over competing solutions was carried out, with the employment of diverse ML algorithms to address the formulated regression tasks. These include Decision Tree (DT), Support Vector Machine (SVM), and Multi-layer Perceptron (MLP) regressors implemented by using the Scikit-Learn Python library [26], as well as the CatBoost regressor from the Catboost Python library [27]. MLP is a feed-forward Neural Network, while CatBoost is an ensemble model that combines multiple DT base learners.

Next, we define the strategy that is applied to train, optimize, and validate the performance of the proposed and the competing ML algorithms, which comprised the steps listed below.

  1. 1.

    Train – test data partitions

Our training and testing data partitions were built using a random selection of points from each of the available Point Clouds. Specifically, we selected 100 points per Point Cloud randomly and without replacement for the training dataset. To comprehensively test the model’s performance, we generated five separate testing datasets, each containing 50 points per Point Cloud, also selected randomly and without replacement. The rationale behind this approach was to obtain a more unbiased assessment of model performance and gain an understanding of the variance in performance scores across different testing datasets.

  1. 2.

    Performance assessment

In assessing the performance of our models for the given regression tasks, we used Mean Absolute Error (MAE) and R-Squared (R2) as our chosen metrics. Importantly, the model optimization process was primarily driven by the MAE score. Performance was separately assessed for each axis, thereby allowing us to individually tune and optimize performance per axis.

  1. 3.

    Model training & fine-tuning

Each regression algorithm was provided with a broad hyper-parameter grid for model training and fine-tuning. This was followed by a Randomized Grid Search combined with fivefold cross-validation on the training data partitions. This process allowed us to identify the optimal model of each type based on the mean validation scores, primarily using the MAE score. This randomized grid search helped us explore a wide range of possible parameters and their combinations to find the most effective ones.

  1. 4.

    Model deployment

All models were then deployed on the testing datasets. Importantly, these datasets were unseen by the models during the training and optimization phases, safeguarding against overfitting and information leakage. The final performance scores were calculated as the mean scores plus or minus the standard deviation (mean ± std) derived from the 5 testing data partitions. This strategy gave us a robust and reliable measurement of the model’s ability to generalize to new, unseen data.

3.7 Generation of Explanations

Post-hoc explainability techniques refer to methods used to interpret the predictions or decisions of machine learning models after they have been trained. These techniques aim to provide insights and explanations for the decisions made by the model to users. Specifically, we used SHAP (SHapley Additive exPlanations) [20], a method specifically designed to measure the impact of input features on the predictions of a machine learning model. The SHAP method is rooted in cooperative game theory, providing a unified measure of feature importance that allocates the “contribution” of each feature to the prediction for each instance in a principled, theoretically sound manner. SHAP values quantify the change in the expected model prediction when conditioning on that feature, reflecting the feature’s contribution to the prediction. They can handle interactions between features and provide fairness attributes because they sum up to the difference between the prediction for the instance and the average prediction. For individual predictions or instances (local interpretability), SHAP values indicate how much each feature in the data set contributes, either positively or negatively, to the target variable. In essence, the SHAP method provides a more detailed view of how specific values of features, whether low or high, impact a prediction by nudging it above or below the average prediction value. This localized insight is aggregated to provide a global understanding of the model, i.e., which features are most important across all instances.

Regarding the integration of XAI into our work, we primarily used the SHAP method to interpret our model’s predictions and decisions. After the model was trained and optimized, we computed SHAP values for all features over the entire dataset. This gave us an overall picture of how each feature contributes to the model’s decision-making process. In addition, we generated SHAP plots to visually illustrate the impact of individual features on the model’s predictions, making it easy to understand the relationships between feature values and their effects on the model’s output. This way, the SHAP method provided us with an interpretability tool that helped in understanding the model’s complex decision-making process, promoting transparency and trust in the model’s outcomes. This utilization of XAI ensured that our model was not just a black box producing predictions, but a comprehensible system where both its decisions and processes are understandable and explainable.

4 Results

4.1 Model Performance – Quantitative Analysis

Tables 1, 2 and 3 showcase the performance of the proposed PointNet-based approach compared to the other machine learning models. PointNet outperformed all other models, achieving the best MAE across all three axes (X, Y, Z) and the highest R2 score on the Y axis. The second-best performance was demonstrated by the MLP model, which achieved the highest R2 scores on the X and Z axes. However, the remaining models performed poorly in comparison. These results can likely be attributed to PointNet’s inherent design to manage point clouds, which renders it superior to other models that do not consider the adjacency of data points.

Table 1 Model performance for X-axis errors
Full size table
Table 2 Model performance for Y-axis errors
Full size table
Table 3 Model performance for Z-axis errors
Full size table

4.2 Qualitative Analysis – Error Maps

Figures 3, 4 and 5 illustrate the predicted deviations of the data points, presenting a distinct 3D deviation map for each axis. Both predicted and actual deviations are represented in each figure for all three axes. These figures effectively highlight the proficiency of the proposed model in predicting the measurement accuracy (deviation) of the scanning device, with the predicted deviations closely mirroring the actual ones. However, a noticeable discrepancy between the predicted and actual deviations is only discernible in the case of the Z axis, corroborating the findings from Tables 1, 2 and 3. Overall, the consistency between the predicted and actual deviations underscores the efficacy of the proposed model, demonstrating its potential to be a reliable tool for estimating the accuracy of scanning devices. Even in the case of the Z axis, where slight deviations occur, the model provides insightful data that could lead to further improvements and enhanced predictability in the next iterations.

Fig. 3
Two 3 D surface graphs of Y versus X and Z for actual and predicted deviations on the X plane. The downward parabolic curve is plotted in the y z plane. The center region of the curve has the highest value. The gradient scale ranges from negative 0.04 to 0.04.

Predicted versus actual deviation maps for the X-axis

Full size image
Fig. 4
Two 3 D surface graphs of Y versus X and Z for actual and predicted deviations on the Y plane. The downward parabolic curve is plotted in the plane. The corner region of the curve has the highest value. The gradient scale ranges from negative 0.03 to 0.04.

Predicted versus actual deviation maps for the Y-axis

Full size image
Fig. 5
Two 3 D surface graphs of Y versus X and Z for actual and predicted deviations on the Z plane. The downward parabolic curve is plotted on the plane. The surface of the curve has the highest value. The gradient scale ranges from negative 0.10 to 0.10.

Predicted versus actual deviation maps for the Z-axis

Full size image

4.3 Explanations

Figures 6, 7 and 8 feature the SHAP summary plots of the trained PointNet model, presented individually for each of the three axes. Features appear on the y-axis, ordered by their global importance in the model’s output. SHAP values are measured on the x-axis, so that each point in the summary plot corresponds to a SHAP value for a given sample and feature. The magnitude of feature values is color-coded from blue to magenta, corresponding to low and high values, respectively, so that the effect of different feature values on the model’s output can be deduced by the summary plot. More specifically, the summary plot is indicative of how feature values (high/low) push individual predictions upward or downward with respect to the average prediction, which is specified by the vertical line at SHAP value zero.

Fig. 6
A SHAP summary plot for the measured data on the X axis. The plots for N x, orientation X, A c mos j, N z, R s, View Ang, inc, orientation Z, orientation Y, N y, orientation Y c mos, direction density, exposure time, and lateral density are distributed horizontally from negative 4 to 6.

SHAP summary plots of the trained PointNet model for the X-axis

Full size image
Fig. 7
A SHAP summary plot for the measured data on the Y axis. The plots for orientation c mos, orientation Y, N y, view Ang, R s, N x, orientation X, N z, inc, orientation Z, ang, lateral density, direction density, exposure time, and A c mos j are distributed horizontally from negative 4 to 6.

SHAP summary plots of the trained PointNet model for the Y-axis

Full size image
Fig. 8
A SHAP summary plot for the measured data on the Z axis. The plots for R s, N z, inc, orientation X, N x, orientation Y c mos, orientation Z, N y, orientation Y, view ang, A c mos j, direction density, ang, lateral density, and exposure time are distributed horizontally from negative 6 to 6.

SHAP summary plots of the trained PointNet model for the Z-axis

Full size image

The most important features affecting the measurement errors of the predicted point on the X-axis are found to be the x-component of surface orientation (Nx) and the x-component of the vector difference between laser and surface orientations (oriX). For both of these features, low values correspond to outputs lower than the average prediction (i.e., toward negative measurement errors), while outputs larger than average (i.e., toward positive errors) are induced by high feature values. Points with low Nx values are those facing the negative side of the X-axis, while large Nx corresponds to points facing the positive side. Taking into account the calculation formula (Eq. 13), it follows that points on both sides are captured inward to the surface, with the larger measurement errors corresponding to parts of the object where the surface is oriented toward the x-axis (lateral direction), scattering light away from the CMOS sensor.

Regarding predicted point measurement errors in the Y-axis, these are mostly affected by the Y-components of vector differences between surface and CMOS orientation (oriYcmos) and surface and laser orientation (oriY). Large values for these features correspond to points mostly facing the positive side of the Y-axis, while low values indicate points mostly facing the negative side of the axis. Considering the calculation formula for measurement errors (Eq. 14), these correspond to points being misplaced inward to the surface. The effect is most visible in areas of the surface facing toward (positive or negative) the Y-axis (scan direction), where most of the incident light is scattered away from the CMOS sensor due to large reflecting angles.

Predicted measurement errors in the Z-axis are highly affected by the measurement distance (Rs) between each point and the rotary head of the scanner, where the laser source and CMOS camera are located. Considering the calculation formula (Eq. 15) and the fact that the objects under study lie in the positive Z-coordinate range, we find that points at large measurement distances are misplaced outward to the surface (negative errors), while points closer to the laser and CMOS (low measurement distance) tend to be captured inward to the surface (positive errors).

Overall, we observe that the model’s response in the prediction of errors along each axis is affected by relevant properties of the geometric setup that match optics intuition.

5 Discussion

5.1 Discussion on Experimental Results

This study presents an innovative approach using a PointNet-based model to handle point cloud data with a focus on predicting measurement accuracy in scanning devices. Unlike traditional machine learning models, the design of the proposed approach is inherently equipped to manage point clouds, accounting for the adjacency of data points, making it ideal for this specific task. Leveraging the unique capabilities of the PointNet-based model, our research aimed to develop a more precise and reliable tool for estimating the accuracy of point cloud scanning devices, an area of increasing importance in numerous fields such as robotics, geospatial sciences, and architecture.

The results of our study demonstrate the superior performance of the PointNet-based approach compared to traditional machine learning models. Our quantitative analysis revealed that the PointNet-based model achieved the lowest MAE on all three axes (X, Y, Z) and attained the highest R2 score on the Y axis, reflecting its enhanced predictive accuracy. The qualitative analysis, in the form of 3D deviation maps, visually underscored the proficiency of the model in predicting deviations. These maps revealed a significant consistency between the predicted and actual deviations, validating the model’s effectiveness. However, a slight discrepancy observed on the Z axis highlights an area for potential improvement. Despite this, even the deviation on the Z-axis offers valuable insights that can be used to refine future iterations of the model, ultimately improving its predictability and reliability.

The SHAP summary plots, represented in Figs. 6, 7 and 8, play a crucial role in explaining how the PointNet-based model operates, highlighting that the model’s predictions are largely guided by key geometric properties consistent with principles of optics. Specifically, measurement errors on the X-axis are influenced by the surface’s orientation and its vector difference with the laser, while Y-axis errors are significantly impacted by the surface’s vector differences with the CMOS and laser orientation. For the Z-axis, the model recognizes that errors are largely dependent on the measurement distance between the point and the scanner’s rotary head. This ability of the proposed model to incorporate and leverage these geometric and optical principles when making predictions not only underscores its robustness but also provides a high degree of explainability that enhances its practical utility in the field of metrology.

5.2 Limitations of the Analysis

This study presents some limitations, mainly regarding the data utilized. We opted to demonstrate the models’ efficiency using spherical geometrical objects, which embody diverse conditions in terms of angles between the laser orientation, surface orientation, and CMOS viewing orientation. However, to assess the generalizability of the proposed approach, the analysis should be extended to include non-spherical geometrical objects. The key point here is that this experimentation requires calibrated objects, rendering the data collection process both costly and time-consuming. Looking ahead, we plan to expand our research to incorporate additional geometries, including both convex and concave structures, to further challenge and refine our model’s capability. By considering diverse geometrical shapes, we aim to enhance the model’s robustness and ensure its applicability across a wider array of scenarios. This extension, though demanding in terms of resources, would enable a more comprehensive understanding of the model’s performance, contributing significantly to the development of more accurate and versatile scanning devices in the future.

5.3 Practical Implications and Future Perspectives

The application of AI in metrology, particularly with the advent of XAI, holds the potential to dramatically enhance both the accuracy and efficiency of measurements, a revolution with profound implications for sectors such as aeronautics and automotive. The integration of ML algorithms can improve precision by identifying patterns and correcting errors within large datasets, consequently fostering automation and promoting real-time data analysis. This approach can result in advanced decision-making, where metrologists are empowered with data-driven insights and correlations, potentially reducing manual interventions significantly. XAI is poised to bring a new layer of transparency and interpretability to these intricate AI systems, providing users with a clearer understanding of the decision-making processes of these models, thus encouraging trust and validation in AI-driven measurements. The use of XAI can also allow for the identification of biases or errors within AI models, promoting a deeper understanding of the measurement process as a whole. This transparency will enhance the credibility, accountability, and reliability of AI in metrology, reinforcing confidence in its application.

Looking forward, we envisage that AI will facilitate adaptive calibration and compensation by dynamically adjusting measurement systems to cater to environmental changes and complex geometries. XAI, in particular, is expected to provide significant advancements in the metrology sector by enabling professionals to gain deeper insights into the inner workings of AI models. This understanding will facilitate data-driven decision-making and allow for potential error sources to be identified and mitigated, promoting continuous improvement in the metrological process. Furthermore, XAI’s role in knowledge transfer, specifically in assisting junior metrologists to understand and validate results, will be instrumental in achieving higher accuracy, repeatability, and reliability in measurements, ultimately leading to zero-defect manufacturing. Companies focused on metrology products and services, like Unimetrik and Trimek, could significantly benefit from XAI’s ability to support decision-making during measurement plan definition, enabling time-saving and error reduction, particularly for less experienced professionals. The adoption of XAI in metrology is thus poised not only to improve process efficiency but also to enhance understanding of the metrological process by providing professionals insights into AI models and the impact of key configuration parameters on measurement accuracy.

6 Conclusions

This chapter effectively demonstrated the superiority of a PointNet-based model in predicting 3D scanning device accuracy, significantly outperforming traditional machine learning models. The unique 3D deviation maps contributed greatly to the model’s explainability, offering clear insight into its decision-making process and resultant errors. Despite a minor discrepancy noted on the Z axis, this study represents a promising advancement in metrology, demonstrating the potential of machine learning models to improve the accuracy, reliability, and explainability of measurements in industries that rely on 3D scanning technology. The consequent benefits have broad implications, spanning from precision manufacturing to preserving archaeological artifacts, bringing about a new era of improved accuracy and accountability in metrology.