Robust affine registration method using line/surface normals and correntropy criterion

Abstract

The problem of matching point clouds is an efficient way of registration, which is significant for many research fields including computer vision, machine learning, and robotics. There may be linear or non-linear transformation between point clouds, but determining the affine relation is more challenging among linear cases. Various methods have been presented to overcome this problem in the literature and one of them is the affine variant of the iterative closest point (ICP) algorithm. However, traditional affine ICP variants are highly sensitive to effects such as noises, deformations, and outliers; the least-square metric is substituted with the correntropy criterion to increase the robustness of ICPs to such effects. Correntropy-based robust affine ICPs available in the literature use point-to-point metric to estimate transformation between point clouds. Conversely, in this study, a line/surface normal that examines point-to-curve or point-to-plane distances is employed together with the correntropy criterion for affine point cloud registration problems. First, the maximum correntropy criterion measure is built for line/surface normal conditions. Then, the closed-form solution that maximizes the similarity between point sets is achieved for 2D registration and extended for 3D registration. Finally, the application procedure of the developed robust affine ICP method is given and its registration performance is examined through extensive experiments on 2D and 3D point sets. The results achieved highlight that our method can align point clouds more robustly and precisely than the state-of-the-art methods in the literature, while the registration time of the process remains at reasonable levels.

Introduction

Today, point set/cloud registration is a noteworthy problem since it contributes to various fields such as computer vision, pattern recognition, image processing and classification (see in [1,2,3,4]), medical imaging and diagnosis (see in [5]), 3D modeling and construction (see in [6,7,8]), machine learning (see in [9]) and other engineering fields. The point cloud registration problem can be defined as finding the spatial transformation between two point sets by aligning them in a space. Furthermore, with the proliferation of 3D LIDARs, point cloud registration has gained importance for mapping and localization problems in robotics (see [10,11,12] studies).

As explained in [13], point cloud registration can be categorized into optimization-based and learning-based registration methods. Currently, learning-based registration techniques are very popular, for example, a global registration method based on deep-learning has been developed in [14], and a deep outlier elimination method for the correspondence determination/rejection stage of registration via spatial consistency has also been presented in [15]. In addition, Huang et al. tried to solve the point cloud registration problem with low overlap ratio using learning techniques [16], and RGB color and depth information was also considered in the feature extraction process of registration by Wan et al. [17]. However, the main drawbacks of such approaches are the need for large training sets, unexpected performance loss in unknown scenes, and no guarantees of similar performance outside the training area [13].

The iterative closest point (ICP) algorithm, which is one of the most common optimization-based registration approaches, can be generalized and guarantee convergence owing to its powerful mathematical infrastructure. For this reason, ICP-like approaches are used together with learning-based approaches during the refinement stage to improve registration performance. For example, it has been observed by [14] that the registration accuracy has increased approximately seven times with ICP refinement, which once again reveals the importance of optimization-based registration approaches.

From a different perspective [18], the registration problem can be roughly divided into two main sub-branches in terms of transformation types; these are linear and non-linear registration. Rigid registration methods, a subset of linear approaches, only have satisfactory registration performance under combinations of rigid motions such as translation, reflection, and rotation movements. On the other hand, affine registration methods can both solve rigid registration problems and overcome the shear mapping and scaling effects.

The rigid ICP, which aligns 3D shapes to estimate the relation between them in terms of a rotation matrix and a translation vector, is developed by [19]. Several enhanced variants of the traditional ICP method have been presented, including point-to-line ICP by [20], generalized ICP by [21], globally optimal ICP by [22], and other variants available in the review study by [23]. To make ICP suitable for linear but not rigid registration problems, the ICP variants such as scale ICP by [24, 25] and affine ICP by [26, 27] have been proposed. However, these ICP variants are not robust to noises, outliers, and/or deformations (see [28, 29] studies) since the least-square criterion is used to construct objective functions to be optimized.

It is obvious that such effects (noises, outliers, etc.) are a fact of real-world applications, and successful and accurate registration under these effects is very important. That is why, robust ICP algorithms including correntropy-based robust ICP by [30], bidirectional correntropy-based robust ICP by [31], probabilistic ICP by [32], robust scale ICP by [33,34,35], robust affine ICP by [29] are proposed for linear registration, on the other hand, a cluster correspondence-based fast method by [18], expectation-maximization and/or Gaussian mixture model(GMM)-based robust methods by [36,37,38], coherent point drift(CPD)-based robust approaches by [39, 40] are developed for non-linear registration.

In studies where variants of the linear registration problem such as rigid, scaled rigid, and affine registration are solved with the use of the correntropy criterion, it is observed that the robustness and matching performance are quite well. Correntropy is a similarity measure between two random variables and by considering model and data sets as random variables, it is possible to eliminate outliers and/or noises better and to find the transformation between the point clouds more precisely.

In this study, we propose a robust affine registration method that combines line normals in 2D space, surface normals in 3D space, and correntropy criterion. The proposed registration algorithm is novel since other robust ICPs available in the literature, which employ correntropy criteria for linear registration problems, utilize point-to-point metrics to construct objective functions. The closed-form solution of the presented approach is achieved for both 2D and 3D registration problems. The extensive experiments show the superiority of the developed method in terms of accuracy rate and efficiency over the CPD method by [39], traditional ICP algorithms developed for affine registration by [26], and robust affine ICP approach by [29]. Although the performance and matching precision increase with the developed method, the elapsed time for each registration stays at tolerable levels.

The rest of this study is organized as follows: in the next section, preliminary information on the linear registration problem is given in detail. The proposed robust affine registration method based on line/surface normal and correntropy criterion is explained in the third section. The experiments are conducted on registration of 2D and 3D well-known shapes and the results achieved that validate the accuracy, robustness, and efficiency of the method developed are shown in the fourth section. The penultimate section includes some discussions and comments on the method developed and the results achieved. The concluding remarks and future work suggestions are stated in the last section.

Overview of linear registration

In the registration problem using feature-based similarity measure given in [41] as

$$\begin{aligned} \min _{j(i)\in \{1,2,...,N_m\}}\sum _{i=1}^{ N_p } \Vert {\mathcal {T}}\left( \mathbf {p}_i\right) -\mathbf {m}_{j(i)}\Vert ^2, \end{aligned}$$

(1)

${\mathcal {T}}\left( \cdot \right)$ term represents the linear or non-linear transformation between the n-dimensional point clouds to be registered defined as model set $M\triangleq \{ m_{i} \}_{i=1}^{ N_m }$ and data set $P\triangleq \{ p_{i} \}_{i=1}^{ N_p }$. To construct the objective function, the corresponding point of $\mathbf {p}_i$, the $i^\mathrm{th}$ element of the data set, in model set $\mathbf {m}_{j(i)}$ is utilized.

Rigid registration

In the simplest form of linear registration, as described in [30], the expression in (1) turns into

$$\begin{aligned} \min _{\mathbf {R},\mathbf {t},j(i)\in \{1,2,...,N_m\}}\sum _{i=1}^{ N_p } \Vert (\mathbf {R}\mathbf {p}_i+\mathbf {t})-\mathbf {m}_{j(i)}\Vert ^2 \end{aligned}$$

(2)

and this is called as rigid registration. The transformations between point clouds including only combinations of rigid motions such as rotation, translation, and reflection can be determined by utilizing many methods available in the literature, but traditional ICP developed by [19] is one of the most preferred methods. By minimizing the objective function in (2), traditional ICP computes the transformation between point clouds in terms of a rotation matrix ($\mathbf {R}$) and a translation vector ($\mathbf {t}$).

ICP for affine registration

The ICP variant, which aims to express linear but not rigid transformation between point clouds by an affine relation, was proposed by [26] as

$$\begin{aligned} \min _{\mathbf {A},\mathbf {t},j(i)\in \{1,2,...,N_m\}}\sum _{i=1}^{ N_p } \Vert (\mathbf {A}\mathbf {p}_i+\mathbf {t})-\mathbf {m}_{j(i)}\Vert ^2. \end{aligned}$$

(3)

Here, affine ICP uses an affine matrix ($\mathbf {A}$) and translation vector ($\mathbf {t}$) to describe the current transformation between point clouds and this method can even handle scaling and skew (shear) effects.

Least-square and correntropy criteria

In general, ICP algorithms consider the point cloud registration problem as a least-square minimization problem. Using this idea, the difference between the transformed data set and the model set is tried to be minimized. Another approach, utilization of correntropy criterion, aims to find the transformation that maximizes the similarity between point sets. Correntropy combines correlation term in probability and entropy term in mechanics and is defined as a similarity measure between two random variables ${\mathcal {X}}$ and ${\mathcal {Y}}$. The mathematical description of the correntropy is given in [42] as

$$\begin{aligned} \mathbf {V}_\sigma ({\mathcal {X}},{\mathcal {Y}})=\mathbf {E}\left[ \kappa _\sigma \left( {\mathcal {X}}-{\mathcal {Y}}\right) \right] \end{aligned}$$

(4)

where $\mathbf {E}\left[ \cdot \right]$ denotes expectation in probability and $\kappa _\sigma \left( \cdot \right)$ represents a symmetric positive definite kernel with $\sigma$ kernel width. For the Gaussian Kernel, Wu et al. [42] is achieved the correntropy definition as

$$\begin{aligned} \mathbf {V}_\sigma ({\mathcal {X}},{\mathcal {Y}}) = \mathbf {E}\left[ \dfrac{1}{\sqrt{2\pi }\sigma }\exp \left( -\dfrac{({\mathcal {X}}-{\mathcal {Y}})^2}{2\sigma ^2}\right) \right] . \end{aligned}$$

(5)

Affine ICP with correntropy criterion

The definition of correntropy criterion in (5) was used by [29] to generate a robust affine ICP variant. The new objective function is constructed as

$$\begin{aligned} \max _{\begin{matrix} det(\mathbf {A})\ne 0, \mathbf {t}, \\ j(i)\in \{1,2,...,N_m\} \end{matrix}}\sum _{i=1}^{ N_p }\exp \left( -\dfrac{\Vert (\mathbf {A}\mathbf {p}_i+\mathbf {t})-\mathbf {m}_{j(i)}\Vert ^2}{2\sigma ^2}\right) \end{aligned}$$

(6)

where the transformation between the model and data sets is computed through maximum correntropy criterion instead of least-square-based minimization. In the same study (see [29] study), the researchers showed that ICP variants using the correntropy criterion perform better than those utilizing the least-square criterion, especially when the point clouds contain noises and/or outliers. That is why, the robust affine point cloud registration method, which we propose in this study, uses the correntropy criterion.

Affine ICP with line/surface normals and correntropy criterion

In all methods mentioned so far, the point-to-point metric is used while deriving the cost function, which takes into consideration the distance from one point to another (see in Fig. 1a). With line/surface normals, it is also possible to derive methods that consider the distance from a point to the defined line or plane (see in Fig. 1b, c). [20] proposed a variant of traditional ICP that uses point-to-line metric to match 2D point sets. However, affine ICP variant using the correntropy criterion and line/surface normals is not available in the literature. As a result, this improved version of affine ICP is introduced in this study to solve the affine registration problem more robustly for both 2D and 3D point clouds.

Fig. 1

Visual demonstrations of (a) point-to-point, (b) point-to-line, (c) point-to-plane metrics and (d) not desired configuration for point-to-line metric case

The objective function using line/surface normals and correntropy criterion can be constructed as

$$\begin{aligned} J_{PL}^{corr}(\mathbf {A},t)= \sum _{i=1}^{ N_p }\exp \left( -\dfrac{\left( \mathbf {n}_{i}^T\cdot \left[ (\mathbf {A}\mathbf {p}_i+\mathbf {t})-\mathbf {m}_{j(i,1)}\right] \right) ^2}{2\sigma ^2}\right) \nonumber \\ \end{aligned}$$

(7)

where $\mathbf {n}_{i}$ is i^th normal vector and $\mathbf {m}_{j(i,1)}$ first correspondence of $\mathbf {p}_i$ in the model set. The superscript ‘corr’ represents the correntropy criterion and ‘PL’ stands for Point-to-Line or Point-to-pLane metrics utilized based on dimension of point clouds to be registered.

Computation of affine transformation

ICP algorithms find closest points iteratively and for the k^th iteration of an ICP variant two steps given as follows are solved:

• 1. Finding corresponding points: To define a tangent line in 2D space, at least two nearest points of $\mathbf {p}_i$, first and second correspondences ($\mathbf {m}_{j(i,1)}$ and $\mathbf {m}_{j(i,2)}$), are required. On the other hand, three corresponding points ($\mathbf {m}_{j(i,1)}$, $\mathbf {m}_{j(i,2)}$ and $\mathbf {m}_{j(i,3)}$) are necessary to find the tangent plane of a surface in 3D space. These correspondences can be determined using the previous transformation $\left( \mathbf {A}_{k-1}, \mathbf {t}_{k-1}\right)$ via many methods available in the literature such as Delaunay tessellation of [43], k-d trees of [44], probabilistic correspondence method of [5]. In this study, the k-d trees approach is applied to assign corresponding points of $\mathbf {p}_i$ in the model set.

• 2. Computing transformation: The affine transformation maximizing the similarity between point clouds in k^th iteration, $(\mathbf {A}_{k},\mathbf {t}_{k})$, can be calculated by maximizing the objective function given in (7). The derivation-based solution procedure of this optimization problem will be explained further.

This iterative process continues until it reaches the predefined maximum number of iterations, or stops if the difference between the current and last transformations remains within a sufficiently small boundary.

Computation procedure for 2D point clouds

In an affine transformation $\left( \mathbf {A}_{n\times n}, \mathbf {t}_{n\times 1}\right)$, the subscripts of matrix $\mathbf {A}$ and vector $\mathbf {t}$ represent the dimensions of them and are related to the dimension of point clouds (n). In the 2D workspace (for $n=2$), the unknown parameters in $\mathbf {A}$ and $\mathbf {t}$ can be gathered in a vector $\mathbf {x}=\left[ x_1, x_2, x_3, x_4, x_5, x_6\right] \triangleq \left[ \mathbf {a}_{11}, \mathbf {a}_{12}, \mathbf {a}_{21}, \mathbf {a}_{22}, \mathbf {t}_x, \mathbf {t}_y \right]$ where $\mathbf {a}_{ij}$ is the elements of matrix $\mathbf {A}$ at the $i^{th}$ row and $j^{th}$ column. With the usage of ${\mathbb {P}}_i$ definition

$$\begin{aligned} {\mathbb {P}}_i= \begin{bmatrix} \mathbf {p}_i^T &{}\quad \mathbf {{0}} &{}\quad 1 &{}\quad 0\\ \mathbf {{0}} &{}\quad \mathbf {p}_i^T &{}\quad 0 &{}\quad 1 \end{bmatrix} \end{aligned}$$

and vector $\mathbf {x}$, the objective function in (7) can be rewritten as

$$\begin{aligned} J(\mathbf {x})= \sum _{i=1}^{ N_p }\exp \left( -\dfrac{\left( {\mathbb {P}}_i \mathbf {x}-\pi _i\right) ^T C_i \left( {\mathbb {P}}_i \mathbf {x}-\pi _i\right) }{2\sigma ^2}\right) \end{aligned}$$

(8)

where $C_i$ is equal to $\mathbf {n}_i \cdot \mathbf {n}_i^T$, and $\pi _i$ stands for $\mathbf {m}_{j(i,1)}$. To calculate optimal $\mathbf {x}$ that maximizes the objective function in (8), derivation-based maximization methods can be utilized such that $\dfrac{\partial J(\mathbf {x})}{\partial \mathbf {x}} = 0\rightarrow \mathbf {x} = \mathbf {x}^*$. If this derivation is applied on the extended version of the objective function as follows:

$$\begin{aligned} J(\mathbf {x})= \sum _{i=1}^{ N_p }\exp \left( -\dfrac{ \mathbf {x}^T{\mathbb {P}}_i^T C_i {\mathbb {P}}_i \mathbf {x} + \pi _i^T C_i \pi _i - 2\pi _i^T C_i {\mathbb {P}}_i \mathbf {x}}{2\sigma ^2}\right) , \end{aligned}$$

the expression

$$\begin{aligned} \sum _{i=1}^{ N_p }\left[ \mathbf {x}^T\left( {\mathbb {P}}_i^T C_i {\mathbb {P}}_i + \left( {\mathbb {P}}_i^T C_i {\mathbb {P}}_i\right) ^T \right) -2\pi _i^T C_i {\mathbb {P}}_i \right] \Delta _i = 0\nonumber \\ \end{aligned}$$

(9)

is obtained in a compact form. Here, $\Delta _i$ corresponds to the exponential part of the objective function which is $\exp \left( -\dfrac{ \mathbf {x}^T{\mathbb {P}}_i^T C_i {\mathbb {P}}_i \mathbf {x} + \pi _i^T C_i \pi _i - 2\pi _i^T C_i {\mathbb {P}}_i \mathbf {x}}{2\sigma ^2}\right)$. The expression in (9) can be arranged as

$$\begin{aligned} \sum _{i=1}^{ N_p }\left( \mathbf {x}^T {\mathbb {P}}_i^T C_i {\mathbb {P}}_i -\pi _i^T C_i {\mathbb {P}}_i \right) \Delta _i = 0 \end{aligned}$$

(10)

since $C_i$ is a symmetric matrix. As a result, using $\mathbf {H} \triangleq \sum _{i=1}^{ N_p } {\mathbb {P}}_i^T C_i {\mathbb {P}}_i \Delta _i$ and $\mathbf {g} \triangleq \sum _{i=1}^{ N_p }\pi _i^T C_i {\mathbb {P}}_i \Delta _i$ definitions, the optimal $\mathbf {x}$ is achieved as

$$\begin{aligned} \mathbf {x}^* =\left( \mathbf {g} \cdot \mathbf {H}^{-1} \right) ^T. \end{aligned}$$

(11)

Here, the necessary terms to construct affine transformation matrix $(\mathbf {A}, \mathbf {t})$ that maximizes the similarity between the point clouds (model and data sets) are calculated with an assumption that matrix $\mathbf {H}$ is an invertible matrix.

The assumption about the non-singularity of matrix $\mathbf {H}$ is valid except for the cases visually demonstrated in Fig. 1d. This means that the relevant point in the data set is also lying on the line/plane defined using the model set. The points satisfying this condition should be ignored while constructing the objective function to eliminate singularity.

On the other hand, it is obvious that the terms $\mathbf {g}$ and $\mathbf {H}$ used to derive optimal $\mathbf {x}$ also include the term $\mathbf {x}$ in the term $\Delta$, and this indicates that there is an algebraic loop in the solution. However, since ICP is an iterative approach the optimal $\mathbf {x}$ vector can be computed. In each iteration, $\Delta$ is computed using the previous transformation $(\mathbf {A}_{k-1},\mathbf {t}_{k-1})$ to achieve $(\mathbf {A}_{k},\mathbf {t}_{k})$.

Extension to 3D point cloud registration cases

The computation procedure of affine transformation is completely the same for 2D and 3D registration problems. The only difference is that the dimensions of the matrices and vectors such as $\mathbf {x}, \mathbf {n}_i, {\mathbb {P}}_i$ change with respect to the dimensions of the point clouds to be registered. Similar to the 2D case, unknown parameters vector can be constructed as $\mathbf {x}\triangleq \left[ \mathbf {a}_{11}, \mathbf {a}_{12}, \cdots , \mathbf {a}_{33}, \mathbf {t}_x, \mathbf {t}_y,\mathbf {t}_z \right]$ for 3D space. Instead of a tangent line of a curve in 2D, $\mathbf {n}_i$ is defined using the tangent plane of a surface. The dimension of ${\mathbb {P}}_i$ is also extended such that

$$\begin{aligned} {\mathbb {P}}_i^{3D}= \begin{bmatrix} \mathbf {p}_i^T &{}\quad \mathbf {{0}} &{}\quad \mathbf {{0}} &{}\quad 1 &{}\quad 0 &{}\quad 0\\ \mathbf {{0}} &{}\quad \mathbf {p}_i^T &{}\quad \mathbf {{0}} &{}\quad 0 &{}\quad 1 &{}\quad 0 \\ \mathbf {{0}} &{}\quad \mathbf {{0}} &{}\quad \mathbf {p}_i^T &{}\quad 0 &{}\quad 0 &{}\quad 1 \end{bmatrix} \end{aligned}$$

is written as new ${\mathbb {P}}_i$ for 3D space.

Fig. 2

Extract model set from well-known images, bat example: (a) bat image, (b) model set from the image by extracting the edges.

Fig. 3

Data and model sets before registration: apple example, (a) original model set (MS), (b) data set, (c) cropped MS, (d) MS with uniform noise, (e) MS with Gaussian noise, (f) cropped MS with uniform noise, and (g) cropped MS with Gaussian noise

In conclusion, the mathematical expression of the proposed affine registration method and a detailed explanation of the solution procedure are given up to here. In the next section, studies on verification of the proposed method in terms of robustness, success rate, accuracy, etc. will be discussed.

Experimental results

To verify the algorithm proposed in this study in terms of accuracy and robustness, its registration performance is investigated on well-known 2D and 3D shapes. In the following sub-sections, the applied procedure to practice matching experiments is given and the superiority of the developed algorithm over other state-of-the-art algorithms is shown in detail.

AfICP-PL-corr for 2D registration

2D registration experiments are conducted on eight different 2D shapes. In each experiment, the edges of the 2D image are extracted to generate the model set. A model set extraction example for the bat experiment is demonstrated in Fig. 2. The extracted shapes are utilized to construct data set and six different model sets. An example of data and model sets for apple experiments is shown in Fig. 3. Here, to achieve the shape in Fig. 3b, the shape in Fig. 3a which is called the original model set is transformed with an affine relation including rotation, translation, shearing, and scaling effects. The left bottom side of the original model set is deleted to examine the missing data effect on registration and the cropped model set is achieved as in Fig. 3c. A uniform noise is added to the original and cropped model sets to reach the model sets in Fig. 3d, f, respectively. Similarly, the model sets in Fig. 3e, g are formed by adding a Gaussian noise on original and cropped model sets, respectively.

Table 1 shows the structural properties of the 2D point sets generated. The names of the shapes are given in the first column and the number of points in the original 2D shapes is placed in the second column. To generate noisy point sets, 20% of the number of points in the original point set is added based on relevant noise distribution. Similarly, the left bottom of the shapes (about one-eighth) is cropped to construct model sets having missing data. The upper and lower boundaries of the original model set of the shapes are given for y and x-axes in the third and fourth columns, respectively.

Table 1 The number of points in the 2D shapes, and the upper and lower boundaries of the 2D shapes registered in this study

The performance of our algorithm is examined on all model set variants in Fig. 3. The achieved results are compared to traditional affine ICP (AfICP-PP-ls) by [26] and affine ICP with point-to-line metric (AfICP-PL-ls) algorithms to see the superiority of our algorithm under noises and outliers. The experiments are conducted by MATLAB 2019a$\text{\textregistered}$ on a PC with Intel Core i7 Q740 1.73 GHz CPU and 4 GB RAM.

When the model set has no outliers, noises, and/or missing data (original model set), the model and data sets are matched successfully with all algorithms (AfICP-PP-ls, AfICP-PL-ls, and AfICP-PL-corr). However, when changes such as noise insertion and/or cropping (to derive model set having missing data) are applied on the model set, it is seen that our algorithm developed in this study is much more successful than AfICP-PP-ls and AfICP-PL-ls in registration. Two separate experiments showing this for the model sets having missing data and uniform noise and the model sets having missing data and Gaussian noise are given in Fig. 4a, b, respectively.

Fig. 4

Registration results of 2D shapes for model set having missing data and (a) uniform and (b) Gaussian noise: (i) the shapes before registration, registration results of (ii) AfICP-PP-ls, (iii) AfICP-PL-ls and (iv) AfICP-PL-corr (our algorithm)

The results obtained so far are sufficient to show the success of our algorithm, but we made even greater changes on the point sets to force our algorithm further. Changes in point sets are clearly seen in the visuals to be given later on where matching performances of the algorithms are evaluated, but it may be necessary to mention verbally here too. Apple_2 and Apple_3 point sets were obtained by adding a bite to the apple shape and removing the leaf on the apple shape, respectively. A new butterfly shape (Butterfly_2) was derived by removing the antennae of the butterfly and by adding spots on its wings. Similarly, the dog shape with the tail down is modified to tail up to generate a new dog shape (Dog_2). A new dragon shape (Dragon_2) was obtained by softening the neck and chin of the previous dragon shape. On the other hand, the stem of the leaf shape has been removed to construct Leaf_2. While making changes to the tail and ears of the bunny shape to get Bunny Stand_2, half of the bat shape was clipped for Bat_2. As seen in Fig. 5a, even no noise insertion and/or part cropping is applied on the model set, it is seen that other algorithms fail to register the point sets, but our algorithm successfully matches the model and data sets despite these modifications.

Fig. 5

Registration results of 2D shapes for modified model sets: (i) the shapes before registration, registration results of (ii) AfICP-PP-ls, (iii) AfICP-PL-ls and (iv) AfICP-PL-corr (our algorithm)

The matching performances of algorithms on modified point sets having missing data and noises are also investigated, and it is seen in Fig. 5b, c that the algorithm developed in this study registers successfully even under these disturbing effects.

Fig. 6

Registration results of 2D shapes: various model set types—(a) the shapes before registration, registration results of (b) AfICP-PL-corr (our algorithm), (c) AfICP-PP-corr

The correntropy criterion-based method developed in this study (AfICP-PL-corr) so far has only been compared to conventional least-square criterion-based methods. However, Fig. 6 shows the comparison of AfICP-PP-corr (affine ICP with point-to-point metric and correntropy criterion) developed by [42] with our method AfICP-PL-corr in terms of registration performance. The results obtained demonstrate that both algorithms are not affected by undesired factors such as noises, missing data, and/or outliers, and they are able to match the point sets successfully since their objective functions are constructed based on the correntropy criterion. However, in addition to the accuracy of the algorithms finding the transformation between two point sets, if we focus on how long it takes to find the correct transformation, it is seen that our algorithm (AfICP-PL-corr) requires much fewer iterations to achieve the final transformation. An example indicating this is given in Fig. 7. The first column shows the registration of apple shape having no major changes, and after a while, all the algorithms have successfully calculated the correct transformation between the point sets. On the other hand, the bitten apple shape registration result is given in the second column and traditional least-squared criterion-based methods have failed to register. In addition to being successful, our algorithm does this with much less iteration than AfICP-PP-corr.

Fig. 7

For the affine registration algorithms, the number of iterations required to match the shapes

AfICP-PL-corr for 3D registration

Similar to 2D registration tests, well-known 3D shapes are taken into consideration to evaluate 3D affine registration performance of the algorithms. Glira’s 3D point cloud data sets including Dino, Lion, etc. shapes have been mostly utilized in our applications (see [45] study).

Since 3D point clouds contain a lot of data, it is aimed to reduce the number of data in 3D point clouds by down-sampling them without disturbing the general features of the 3D shapes to make registration faster with algorithms such as ICP. Down-sampling methods are not in the scope of this study, but the available 3D down-sampling methods in MATLAB software are directly used to down-sample our 3D point clouds. An example of down-sampling on the 3D watering-can shape can be seen in Fig. 8.

Fig. 8

Original point cloud and its down-sampled versions

Table 2 The number of points in the down-sampled 3D shapes, and the upper and lower boundaries of the 3D shapes registered in this study

The structural properties of the 3D point sets to be used for evaluating the 3D affine matching performances of the algorithms are listed in Table 2. The names of the shapes are given in the first column and the number of points in the original down-sampled 3D shapes registered is placed in the second column. To generate noisy point sets, 10% of the number of points in the original point set is added based on relevant noise distribution. Similarly, the left-front-bottom of the shapes (about one-eight) is cropped to construct model sets having missing data. The upper and lower boundaries of the original model set of the 3D shapes are given for z-, y-, and x-axes in the third, fourth, and fifth columns, respectively.

Fig. 9

Data and model sets in 3D before registration: lion example, (a) original MS, (b) data set, (c) cropped MS, (d) MS with uniform noise, (e) MS with Gaussian noise, (f) cropped MS with uniform noise, and (g) cropped MS with Gaussian noise

Similar to 2D cases, for each 3D registration experiment, the down-sampled 3D shapes are utilized to construct data set and six different model sets. An example of data and model sets is shown in Fig. 9. Here, the shape in Fig. 9a is called the original model set and this is transformed with an affine relation to achieve the data set in Fig. 9b. The left-front-bottom side of the original model set is erased to achieve the cropped model set in Fig. 9c) A uniform noise is inserted into original and cropped model sets to reach the model sets in Fig. 9d, f, respectively. Similarly, the model sets in Fig. 9e, g are created by adding a Gaussian noise on original and cropped model sets, respectively.

In 3D cases, the registration performance of the CPD algorithm was also considered for comparisons. Thus, it has been shown that the method proposed in this study is superior not only to other ICP-based variants but also to different approaches in the literature. CPD approach is an efficient registration method for point sets including noises and/or outliers since it converts the registration problem into probability density estimation by expressing point clouds in terms of the GMM.

Fig. 10

Registration results of 3D dog shape: (a/e-1) original model set vs (a/e-2) model set having missing data and uniform noise - (a) the shapes before registration, registration results of (b) CPD, (c) AfICP-PP-ls, (d) AfICP-PL-ls and (e) AfICP-PL-corr (our algorithm)

Fig. 11

Registration results of 3D shapes: model set having missing data and Gaussian noise (a) the shapes before registration, registration results of (b)CPD, (c) AfICP-PP-ls, (d) AfICP-PL-ls and (e) AfICP-PL-corr (our algorithm)

All algorithms (CPD, AfICP-PP-ls, AfICP-PL-ls and AfICP-PL-corr) can find the correct transformation between model and data sets when there are no disturbing effects such as noise, missing data, etc. on the 3D model set. On the other hand, it is seen that our algorithm is much more successful in registration than CPD, AfICP-PP-ls and AfICP-PL-ls when changes such as noise addition and/or cropping (to generate a 3D model set having missing data) are applied to the model set. The affine 3D registration results of the algorithms, for example, are shown for model sets having missing data and uniform noise in Fig. 10 and for model sets having missing data and Gaussian noise in Fig. 11. Parts that are not correctly matched by CPD, AfICP-PP-ls and AfICP-PL-ls are indicated with arrows for better visualization and perception.

To force the algorithms further, we tested the partial registration performance of them on lion shape-based 3D point clouds in Fig. 12. As given in Fig. 13, even no noise insertion and/or part cropping is applied on the model set, it is seen that other algorithms fail to register 3D point clouds, but our algorithm successfully matches the model and data sets despite these major changes.

Fig. 12

The generated model sets based on 3D point cloud based on lion shape to see partial registration performance of our algorithm—(a) original model set, (b) Partial 1: front section of the lion, (c) Partial 2: top section of the lion, (d) Partial 3: rear section of the lion

Fig. 13

Partial registration performance of the algorithms—3D model sets with different types: MS having uniform noise and missing data in column 1, MS having missing data in column 2, original partial MS in column 3, MS having missing data and down-sampled with random ds method; (a) before registration, (b) CPD , (c) AfICP-PP-ls, (d) AfICP-PL-ls, and (e) AfICP-PL-corr (our algorithm) performances

Numerical results for 2D/3D registration experiments

In addition to visual verification, the error metrics $\varepsilon _{A}=\Vert \mathbf {A}^*-\mathbf {A}\Vert _2$ and $\varepsilon _{\mathbf {t}}=\Vert \mathbf {t}^*-\mathbf {t}\Vert _2$ are defined to compare the registration performance of the algorithms. Here, $(\mathbf {A}^*,\mathbf {t}^*)$ represents the true affine transformation between model and data sets, and $(\mathbf {A},\mathbf {t})$ denotes the affine transformation estimated by the utilized algorithm. For modified model sets cases in 2D experiments, all compared numerical results are drawn in Fig. 14a for matrix $\mathbf {A}$ and in Fig. 14b for translation vector $\mathbf {t}$. The results in the bar plots say that our algorithm is able to register the shapes in each case with reasonable error levels. This proves that our affine registration algorithm is also highly robust to outliers, noise effects, and/or major changes in the point sets.

Fig. 14

a Matrix $\mathbf {A}$ and b translation vector $\mathbf {t}$ estimation error on affine registration of 2D point sets having major changes, A$_2$: Apple 2, A$_3$: Apple 3, B$_2$: Bat 2, BS$_2$: Bunny Stand 2, BF$_2$: Butterfly 2, D$_2$: Dog 2, Dr$_2$: Dragon 2, L$_2$: Leaf 2 shapes—the results for (i) original MS, (ii) cropped MS, MS with (iii) uniform and (iv) Gaussian noise, cropped MS with (v) uniform and (vi) Gaussian noise

The overall 2D and 3D affine registration performances of the algorithms are listed comparatively in Table 3. Here, it is demonstrated that our algorithm (AfICP-PL-corr) has the minimum registration errors in all cases in terms of having noise and/or missing data. Another important outcome of this table is that as the similarity ratio between the model set and the data set decreases, the estimation performance of the algorithms falls.

Success rates for 2D/3D registration experiments

Figure 15 shows the overall success rate, the number of required iterations on average, and the mean elapsed time per registration for affine registration of 2D and 3D shapes. Successful registration is defined as finding affine transformation with an error of less than 10%. On the other hand, the number of required iterations for a match is defined as the iteration where transformation error falls below 1% otherwise equals the maximum number of iterations. The statistics presented in Figure 15 are generated by performing 576 separate experiments in 2D space and 108 separate experiments in 3D space. In affine registration, eight different 2D and four different 3D shapes having various model sets such as original, cropped, uniform or Gaussian noise inserted and/or combinations of them are utilized.

It is deduced that our method is the most successful and highly robust algorithm in 2D and 3D affine registration among the algorithms compared in this study. In 2D registration, AfICP-PP-corr and our algorithm (AfICP-PL-corr), which use the correntropy criterion to construct objective function, have similar registration performance in terms of accuracy (the result of our algorithm is slightly better). However, in 3D registration, the estimation performance of AfICP-PP-corr decreases and it fails approximately 20% of the registration experiments (see in Fig. 15a).

AfICP-PL-ls, another algorithm that uses line/surface normals similar to AfICP-PL-corr, also requires fewer iterations to match the 2D/3D shapes than approaches using the point-to-point metric (see in Fig. 15b). Moreover, as seen in Fig. 15c, it is the fastest algorithm in 3D registration since the correntropy criterion brings extra computational load for AfICP-PP-corr and AfICP-PL-corr. However, rather than being fast, the main thing is the success of the algorithm in matching and its capability to resist real-world factors such as noises/outliers. When Fig. 15a is examined, the results achieved demonstrate that the success rate of AfICP-PL-ls in 2D registration is terrible (around 40% successful matches), on the other hand, its success rate increases in 3D registration, but it cannot even reach success level of AfICP-PP-corr.

Our algorithm is also compared with the CPD algorithm, which is a robust and non-ICP-based registration method, for 3D point set experiments. It is apparent that AfICP-PL-corr is better in terms of not only success rate but also computational requirements. Since the CPD approach utilizes the GMM for representing the point sets, its registration performance is higher for randomly down-sampled point clouds than those down-sampled with grid average technique. However, our method developed in this study can tolerate both of them.

Table 3 The average 2D and 3D affine registration performances of the algorithms (CPD, AfICP-PP-ls, AfICP-PL-ls, and AfICP-PL-corr): the affine transformation estimation errors of the algorithms are shown in terms of $\left( \varepsilon _A, \varepsilon _t \right)$

Fig. 15

For the algorithms used in 2D/3D point cloud registration, (a) the overall success rates of them, (b) the number of iterations required for each registration on average, and (c) mean elapsed time for each registration

Computational complexity analysis for AfICP-PL-corr

The computational load of the proposed approach is analyzed in Fig. 15 in terms of the average elapsed time and the number of iterations required per registration. However, the time needed to match point sets will vary depending on the properties of the hardware on which the algorithm is executed. Therefore, utilizing the ${\mathcal {O}}(\cdot )$ framework, the amount of work the CPU has to do for executing the AfICP-PL-corr algorithm will be explored based on the size of the input point set.

Fig. 16

The computational complexity of the developed AfICP-PL-corr algorithm: the average elapsed time per iteration vs the number of points in the point cloud for (a) 2D and (b) 3D registration, the computation time relationship between the 2D and 3D scenarios

Fig. 17

Registration results of 2D shapes for varying outlier levels: the number of outliers is 10%, 20%, 30%, 50%, and 70% of the number of points in the original model set—(i) the shapes before registration and (ii) registration results of AfICP-PL-corr (our algorithm)

Let n be the number of points in the input point cloud, then the relationship between time elapsed in each iteration and n has been obtained as in Fig. 16a, b for 2D and 3D matching cases, respectively. The achieved linear curves in Fig. 16a, b shows that the computational complexity of the AfICP-PL-corr algorithm is ${\mathcal {O}}(n)$. On the other hand, it can be deduced from Fig. 16c that registration takes less time in 2D space than in 3D space even if the number of points in the sets is identical. There is a constant ratio between the matching times since the 2D and 3D registration curves almost overlap with each other when the time elapsed in 2D is scaled with the constant scalar $s_c = 4.57$. As ${\mathcal {O}}(s_c\times n)$ is equivalent to ${\mathcal {O}}(n)$ according to the properties of the ${\mathcal {O}}(\cdot )$ framework, the results confirm that the computational complexity of the developed algorithm is ${\mathcal {O}}(n)$.

Evaluation of AfICP-PL-corr registration performance for different outlier levels

The superior aspects of the affine ICP approach developed in this study, which uses the line/surface normals and correntropy criterion, have been demonstrated by extensive experiments on 2D and 3D data sets. In this sub-section, it is aimed to investigate how the outlier elimination performance changes for different outlier levels. In this context, various degrees of outliers (10%, 20%, 30%, 50%, and 70%) with both uniform and Gaussian distributions were injected into 2D and 3D point clouds, and in this case, registration tests were performed. Some matching results for 2D and 3D point sets are given in Figs. 17 and 18, respectively. In total, 72 experiments were carried out for 2D and 96 experiments for 3D. It has been observed that the point sets were successfully matched in both two-dimensional and three-dimensional trials.

Fig. 18

Registration results of 3D shapes for varying outlier levels: the number of outliers is 10%, 20%, 30%, 50% and 70% of the number of points in the original model set—(i) the shapes before registration and (ii) registration results of AfICP-PL-corr (our algorithm)

Fig. 19

Numerical comparison for varying outlier levels: 2D registration (top-left) and 3D registration (top-right) error rates, variation of the average number of iterations and time required for each 2D (bottom-left) and 3D (bottom-right) registration attempt according to different outlier levels

The achieved numerical results in Fig. 19 demonstrate that there is no significant relationship between the level of outliers and the registration error level of the method proposed in this study. However, the number of iterations required to match the point clouds, and thus the matching time, increase proportionally with the quantity of outliers.

As a result, the ICP variant proposed in this study, AfICP-PL-corr, is highly successful in affine registration of 2D and 3D point sets. Compared to other algorithms, although it is slow in terms of the time required for each iteration, the total registration duration is relatively shorter as it achieves the result (transformation) with less iteration and its success rate shows that AfICP-PL-corr is quite robust in point set registration.

Discussion

An ICP algorithm tries to estimate the transformation between two point sets by optimizing a cost function iteratively. However, except for special cases such as developed methods by [22, 46], ICP algorithms converge to a locally optimal solution. Therefore, an initial guess is very important for ICP algorithms to ensure that the correct transformation is going to be found. Before performing the point cloud registration process, it makes sense to overcome getting stuck in the local minimum problem by running an initial guess computation algorithm like covariance matrix-based approach by [47] or adding some constraints to the cost function such as bounding rotation angle (see [48, 49] studies). However, since the aim of this study is to show the matching performance of the proposed registration algorithm under effects such as noises, deformations, and/or outliers, an initial guess algorithm was not used in experimental verification tests. By keeping the amount of rotation between the two-point sets registered at low levels, the divergence of the algorithms originating from the initial guess error has been prevented. In all experimental studies, the identity matrix was selected as the initial transformation.

For the algorithms utilized for state-of-the-art comparisons and our algorithm developed in this study, the average elapsed time for each registration is given for 2D and 3D registration cases in Fig. 15c. When these results shown in the time domain are analyzed, it is seen that the matches take quite a long time. These periods are much longer, especially for 3D registrations where the number of points in the point clouds is high. However, it should be noted that the speeds and success rates of the algorithms were evaluated with respect to each other only. As a result, no optimization has been made for the algorithms or no attention has not been paid to coding to make the algorithms faster. Coding was done using basic functions available in the MATLAB environment such as matrix multiplication and inverse operations. In addition, the PC where the tests are performed is insufficient in terms of computing capacity. Currently, it may be thought that this algorithm is not suitable for real-time applications, but it is possible to take some precautions to run the algorithms faster for real-time works.

Conclusions

In this study, a robust affine ICP variant using the correntropy criterion and line/surface normals is presented to solve the affine point cloud registration problem. Correntropy-based affine ICP variants are highly robust to noises, outliers, and missing data effects, and with our implementation of line/surface normals in the construction of objective function, the number of required iterations to find the transformation between point clouds has been reduced and registration performance in terms for accuracy has been increased. Optimization of the objective function defined to compute current transformation is given in detail as a closed-form solution. Our method (AfICP-PL-corr), some other affine ICP variants (AfICP-PP-ls, AfICP-PL-ls, AfICP-PP-corr), and CPD are employed in 2D and 3D point cloud registration experiments for comparison, 594 separate experiments on 8 different shapes in 2D and 132 separate experiments on 4 different shapes in 3D, and it is achieved that our algorithm is better in terms of robustness and accuracy rate. As shown, our algorithm also works well in partial registration problems.

The study, which is given in [12], applied affine ICP for the indoor precise localization problem of mobile robots. In future studies, we would like to deploy this affine ICP variant developed in this study for the localization problem. Moreover, in the current version of our method, the transformation between point clouds is found by locally optimizing the objective function. However, finding a globally optimal solution set is highly important not to diverge. For that reason, any modification and/or enhancement can be deployed on our method such as bidirectional distance extension approach for ICP algorithms or global optimization approaches. Similarly, the initial guess is crucial for the ICP algorithm to make proper estimations. This pre-process can be attached to the method proposed in this study since it is essential especially for successful registration of the point clouds having a high amount of rotation difference between each other.