A 3D Surface Reconstruction Method Based on Delaunay Triangulation

Abstract

Point cloud models acquired by passive three-dimensional reconstruction systems based on binocular or multi-view involve large amounts of noise and its distribution is uneven, which affects the accuracy of surface reconstruction. To tackle the problem, we proposed a three-dimensional surface reconstruction method based on Delaunay triangulation. First, use Delaunay triangulation to get a fully adaptive decomposition of point cloud, then the output triangular mesh was represented using dual graph, so by using graph cut optimization the initial surface model was obtained. Second, the deformation model was used to optimize the initial surface model, and then adopted photometric similarity function and Laplace operator to refine surface details. Finally, the refinement was transformed into an iterative procedure, by which the real surface of the object was accurately approximated. We experimented on four standard datasets of Castle-Entry, Castle, Fountain and Herzjesu. The result showed that compared with Poisson surface reconstruction and floating scale surface reconstruction algorithm, the proposed method was more adaptable and robust to noise and outliers, and the detailed information recovery of local surface reconstruction was better. It showed that the method proposed in this paper effectively improved the accuracy and completeness of surface reconstruction, which could reconstruct high quality three-dimensional surface models of object from the point cloud models with lots of noise and complex topological structures.

Supported by NSFC under Grant 61860206007 and 61571313.

1 Introduction

3D reconstruction technology is widely applied in many fields such as virtual simulation [1, 23], non-contact measurement [21], virtual reality [7], battlefield environment perception [4, 22], etc. Therefore, it is a significant problem to process 3D point cloud acquired by 3D digital scanning devices, and achieve 3D reconstruction effectively and accurately. In this paper, we focus on processing point cloud to restore the geometry and topology structure of objects. Considering that 3D cloud point acquired by passive 3D reconstruction systems based on binocular or multi-view involves large amounts of noise and outliers, which affects results of surface reconstruction, we proposed an approach based on Delaunay triangulation. First, Delaunay triangulation was constructed by incremental insertion, after that output triangular mesh was converted into the representation of dual graph, and then we transform the initial surface extraction problem into graph cut optimization. Then due to the loss of details and the roughness of initial surface model, we optimized the initial surface by minimizing the internal and external energy functions of deformation model globally. Finally, surface model was approximated to the real surface of the object through several iterations of all vertices on the surface.

2 Related Work

Surface reconstruction is a major field in computer vision at present. Many surface reconstruction methods is summarized in [2], in this section, we intend to introduce related approaches to our method, including implicit surface reconstruction approaches [5, 9, 10, 20], and surface reconstruction approaches based on Delaunay triangulation [8, 11,12,13,14,15].

Kazhdan et al. [9] formulates the surface reconstruction from oriented points as a spatial poisson problem, which suffers a tendency to oversmooth the data. Kazhdan et al. [10] modifies the poisson reconstruction algorithm to add incorporate positional constraints, which improves the geometric surface and detail characteristics of the surface. However, reconstruction results of the approach is easily affected by data redundancy. Fuhrmann et al. [5] achieves surface reconstruction by extracting the zero-level set of the implicit function. Their approach could be extended theoretically, but the interpolation ability of the approach is limited due to the small support area. Wang et al. [20] proposes a surface reconstruction approach based on implicit pht-splines. Their approach could reconstruct high-quality results, but which requires high demand of memory.

The visual hull introduced in [13] is unsuitable for outdoor scene reconstruction because of the dependence on separating object from background. Pan et al. [15] extracts surface by using probabilistic tetrahedron carving algorithm. But their approach is not resistant against outliers and inflexible for the batch processing of data. An approach of multi-view reconstruction is proposed in [12]. They define an energy equation based on visibility information and surface parameters, and minimize this energy by graph cuts [3]. Labatut et al. [11] improves the approach in [12], the photo-consistency is replaced by surface quality term in [11] and smoothing term is combined with a purely discrete visibility term. Their approach has good scalability, but it cannot construct weak-support surface well. Jancosek et al. [8] achieves multi-view reconstruction preserving weakly-supported surfaces which works on [11]. Mostegel et al. [14] proposes a scalable approach of surface reconstruction from 3D multi-scale multi-view stereo point clouds. Their approach extracts surface hypotheses from local Delaunay tetrahedralization and merges overlapping surface hypotheses by graph cuts [3], finally a consistent mesh is generated. Their approach is resilient to outliers fragments and vast scale changes, but the results of reconstruction is limited by the localized strategies taken in the original detection process.

3 Initial 3D Surface Reconstruction

In this paper, bundle system [16, 17] is adopted to obtain camera parameters and initial 3D point cloud. The point cloud is processed by PMVS [6], and then we apply Delaunay triangulation to get a fully adaptive decomposition of point cloud, which is constructed by incremental insertion. For the initial dense point cloud, we find the nearest point of the point to be inserted in Delaunay triangulation based on the visibility information of the point, and then the maximum reprojection error between two 3D points is computed. If the reprojection error is above threshold e, the point would be inserted in Delaunay triangulation, or the position of the nearest point would be recomputed. In addition, the visibility information and confidence information of the nearest point would be updated. This step outputs a Delaunay triangulation T through traversing all 3D points.

Fig. 1.

The visibility conflict is the edge interested with ray $c_jp_i$, the arrowed edges are parts of the representation of the dual graph, which weight the edge.

Next we convert the Delaunay tetrahedralization T into the representation of a dual graph and set weight associated with it. Therefore, the surface is extracted by performing graph cut optimization on this dual graph. The detailed procedure is as follows: we assign weights to directed edges of dual graph based on visibility information and smoothing terms. Accordingly, the minimum cut is extracted by solving the energy equation as shown in formula (1). S is the initial reconstruction surface, $E_{visibility}$ is the visibility term, $E_{smooth}$ is the smoothing term and $\lambda$ is the weight coefficient. The energy equation is as follows:

$$\begin{aligned} E(S)=E_{visibility}+\lambda E_{smooth}(S) \end{aligned}$$

(1)

According to the visibility constraints, all vertices v appearing on the final surface S should not be occluded from the viewpoints b they come from and therefore visibility item could be computed as the collision detection problem of the line-face intersection conflict, as shown in Fig. 1. $p_i$ is a 3D point, $c_j$ is the center of camera j. We calculates the intersection of the ray $c_jp_i$ and Delaunay tetrahedralization T. $\lambda _{visibility}$ is the size of visibility information set. ${T_j(p_i)}$ is the credibility of the perspective, which represents the probability that vertex appears on the final surface model. Combining the Gaussian kernel function to reduce noise, the formula is as follows:

(2)

In formula (2), t is the edge in the dual graph and the intersecting triangle of the adjacent tetrahedron in the Delaunay tetrahedralization, p is the vertex in the Delaunay, c is camera center, $(p,c)\cap t$ is the ray pc intersects t, d is the distance from the intersection with $p_i$, $\sigma$ is the standard deviation of Gaussian function. Traversing all vertexes in the Delaunay tetrahedralization, we compute the visibility term by accumulating the directed edges weights as shown in formula (2).

The surface model will be uneven while the energy equation is only computed by $E_{visibility}$. In order to reduce the influence of outliers and noise points and restore the detailed information of the surface model. Triangle t is the intersecting of adjacent Delaunay tetrahedrons. The angle between the triangle plane and the external sphere of two tetrahedrons is ${\alpha }$, ${ \beta }$. The representation of the angle is shown in Fig. 2. While ${\alpha }$, ${\beta }$ is getting smaller, the possibility of cutting edge passing through triangle t is greater, which means the possibility of the cut edge appearing on the final surface is higher. ${\alpha }$ is calculated as.

$$\begin{aligned} \cos (\alpha )=\frac{\overrightarrow{pc}\cdot \mathbf {n}}{|\overrightarrow{pc}|\cdot |\mathbf {n}|} \end{aligned}$$

(3)

$\overrightarrow{pc}$ is the vector from vertex p to sphere centre c. $\mathbf {n}$ is the normal vector of triangle plane, $E_{smooth}$ is computed as:

$$\begin{aligned} E_{smooth}(S)=\sum _{t\in T}1-min\{cos(\alpha ),cos(\beta )\} \end{aligned}$$

(4)

The Smooth term is computed by traversing all the edges of dual graph. The Delaunay tetrahedron is labelled as inside/outside by graph cut optimization [3]. Triangles from label outside to inside are extracted as minimum cut edges and the initial reconstructed surface result S is obtained.

Fig. 2.

It shows the 2D representation of the angle, $\alpha$ is the angle between triangle plane and external sphere of tetrahedrons, $\beta$ is calculated the same way.

4 Surface Optimization Based on Deformation Model

Details are inevitably lost when smoothing the initial surface model. In this section, initial surface model is the input, which is optimized by deformation model to achieve surface optimization. In the deformation model, we adopted photometric similarity function as date term, and the internal energy function of the active contour model is calculated as regularization item by using Laplace operator. As the internal and external energy functions are minimized globally, details of the surface model can be restored and a more refined surface reconstruction model can be obtained. The energy equation of the deformation model is as:

$$\begin{aligned} E(S)=E_{int}(S)+\lambda E_{ext}(S) \end{aligned}$$

(5)

S is the initial surface model, ${\lambda }$ is the weight of data item, ${E_{int}(S)}$ is the regularization item, which is responsible for constraining the topological structure of the surface model to prevent self-intersection and producing over smooth surface in the reconstruction process. ${E_{int}(S)}$ is computed by active contour model. ${E_{ext}(S)}$ is data item, which approximates the surface model to target model. ${E_{ext}(S)}$ is computed by photometric similarity function.

4.1 Data Term

The data term ${E_{ext}(S)}$ in the deformation model is computed by the photometric difference of images as:

$$\begin{aligned} \nabla E_{ext}=\sum _{I_i=0}^n\sum _{I_j=0\cap I_i\ne I_j}^n\nabla M_{I_i,I_j}(S) \end{aligned}$$

(6)

Matching function ${M_{I_i,I_j}(S)}$ is the difference between ${I_i}$ and ${I_j}$, $\prod _j$ is the projection of the camera j, $\prod _{i,S}^{-1}$ is reprojecting ${I_i}$ onto the surface, ${I_j}\circ \prod _j\circ \prod _{i,S}^{-1}$ is reprojecting the ${I_j}$ onto ${I_i}$, $\varOmega _{i,j}^S$ is the reprojection domain. ${M_{I_i,I_j}(S)}$ is as follows:

$$\begin{aligned} M_{I_i,I_j}(S)=M|_{\varOmega _i\cap \varOmega _{i,j}^S}(I_i,I_j\circ \prod _j\circ \prod _{i,S}^{-1}) \end{aligned}$$

(7)

The data item gradient value of surface S is obtained by partial derivative of matching function with respect to variable x as:

$$\begin{aligned} \nabla M_{I_i,I_j}(S)=\int _{\varOmega _i\cap \varOmega _{i,j}^S}\partial M_{I_i,I_j}(x_i)DI_j(x_j)D\prod _{j}(x)\frac{\mathbf {n}^T\delta S(x)}{\mathbf {n}^T\mathbf {d_i}}\mathbf {d_i}dx_i \end{aligned}$$

(8)

x is the reprojection 3D point of ${x_i}$, ${DI_j(x_j)}$ is the gradient of the image, ${D\prod _j(x)}$ is Jacobian matrix of $\prod _j$, $\mathbf {d_i}$ is the vector representing the camera i and the point x, $\mathbf {n}$ is the normal vector of 3D points.

4.2 Photometric Similarity Function

Matching function ${M_{I,J}(S)}$ is constructed by ${L_{I,J}(x)}$ on the basis of the local dependency between image I and image J.

$$\begin{aligned} M(I,J)=\int _{\varOmega _i\cap \varOmega _{i,j}^S}1-L_{I,J}(x_i)dx \end{aligned}$$

(9)

${L_{I,J}(x)}$ is photometric similarity function of image I and J at pixel x, $\varOmega _{i,j}^S$ is the reprojection domain. In ${I_i}$, ${w_i}$ is $n\times n$ pixel block centered on ${x_i}$, ${w_j}$ is $n\times n$ pixel block centered on ${x_j}$. ${g_i}$ is the grayscale value at pixel ${x_i}$, ${\sigma _g}$ is the color standard deviation, ${\sigma _x}$ is the distance standard deviation.

$$\begin{aligned} L_{I,J}(x_i)=\frac{cov(w_i,w_j)}{\sqrt{cov(w_i)cov(w_j,w_j)}} \end{aligned}$$

(10)

$$\begin{aligned} E(x)=\frac{\sum _m^{n\times n}w_mx_m}{\sum _m^{n\times n}w_m} \end{aligned}$$

(11)

$$\begin{aligned} w_m=e^{(-\frac{|g_m-g_i|^2}{2\sigma _g^2}+\frac{|x_m-x_i|^2}{2\sigma _x^2})} \end{aligned}$$

(12)

4.3 Regularization Term

The regularization term is constructed by the internal function of the active contour models to prevent the self-intersection or producing over smooth of the surface model. We simplify the weight, ${\lambda _1}$ is the elasticity, ${\lambda _2}$ is the intensity coefficient to control the curves degree of surface, which ultimately makes the curvature of the surface tend to be stable. The minimum of energy function for equation is solved by Euler-Lagrange theorem as:

$$\begin{aligned} \nabla E_{int}(S)==\lambda _1\triangle S-\lambda _2\triangle ^2S \end{aligned}$$

(13)

${\varDelta }$ is the laplace operator, ${\varDelta ^2}$ is the biharmonic Operator. For each vertex v in the surface S, the laplace operator is computed as formula (14), N(v) is a set of 1-ring nearest vertices of v. n is the number of N(v). The gradient of v is as:

$$\begin{aligned} \triangle v=(\sum _{i\in N(v)}(\frac{v_i}{n}))-v \end{aligned}$$

(14)

$\triangle ^2v$ is calculated as:

$$\begin{aligned} \triangle ^2v=\sum _{i\in N(v)}(\frac{\triangle v_i}{n})-\triangle v \end{aligned}$$

(15)

For all vertices v of the surface, the ${\nabla E_{int}(v)}$ is accumulated by ${\nabla E_{int}(S)}$, as shown in formula (16):

$$\begin{aligned} \nabla E_{int}(S)=\sum _{v\in S}\nabla E_{int}(v) \end{aligned}$$

(16)

4.4 Optimizing of the Final Surface

Finally, the surface is optimized by formula (5), and the discrete form of formula (5) is shown in formula (17). After several iterations of all vertices on the surface by gradient descent, surface model is approximated to the real surface of the object.

$$\begin{aligned} S^{k+1}=S^k+\triangle t(\nabla E_{int}(S)+\lambda \nabla E_{ext}(S)) \end{aligned}$$

(17)

${\lambda }$ is the weight of data term, ${\varDelta t}$ is the incremental step. All vertices v are iterated until the whole surface model converges. Finally, the reconstructed surface model is obtained.

5 Experiments

In this section, experiments are divided into two parts. To confirm the effectiveness of our method, we tested on four standard datasets [18]: Castle-Entry, Castle, Fountain and Herzjesu, and compared the experimental results with PSR [10] and FSSR [5]. Moreover, in order to verify the accuracy of our approach, we compared the reconstructed surface models with real three-dimensional models, and have experimented our method and GDMR on four datasets [18] to evaluate the similarity between the experiment results and ground truth of models.

In experiments, bundle system [16, 17] is adopted to obtain the initial 3D point cloud, and then we process the point cloud by PMVS [6], which is used as the input of the surface reconstruction. In the proposed approach, we get the initial reconstruction surface with parameter ${{\sigma }=l/2}$, where l represents the median of Delaunay side length, after that we optimize the surface by deformation model, and set threshold ${n=4}$, we weight ${{\lambda }_1=0.8}$, ${{\lambda }_2=0.2}$ in optimation energy formulation, the weight factor ${\lambda }$ of the iteration equation is set ${{\lambda }=0.5}$, step ${{\triangle }t}$ at each iterations is set ${{\triangle }t=0.5}$, which is reduced by 0.02 times after each iteration.

For the qualitative evaluation, we conducted on the four standard datasets [18] compared with PSR [10] and FSSR [5]. PSR is experimented at a depth of 11. Octree depth of FSSR is set to 10. As illustrated in Fig. 3, we can see that in the easily occluded areas such as pillars of the Herzjesu, our method reconstruct the surface with more details. In areas with uneven distribution of samples such as windows of the Herzjesu, our method reconstruct smoother wall than the others. In Fig. 4, surface details of spiral sculpture of the Fountain are more accurate and closer to the real model. In Fig. 5, the pillar and edge of Castle-entry are more detailed, while the reconstructed surfaces of PSR and FSSR are smoother and image details are lost. In Fig. 6, the reconstruction results of Castle show that the proposed algorithm is superior to PSR [10] and FSSR [5] in terms of detail recovery and reconstruction integrity.

Figures 3, 4, 5 and 6 demonstrate that our method increases the local details information of reconstructed surface, and it can be found that PSR [10] and FSSR [5] are vulnerable to a large number of noise points and outliers, resulting in poor reconstruction effect. Our method improves the precision of the surface reconstruction due to decomposing the surface model, and obtains accurate surface reconstruction by optimizing the surface details through the deformation model.

Fig. 3.

The reconstruction result experimented on Herzjesu shows that the door and door frame reconstructed by the proposed method is more clearly than the other methods [5, 10].

Fig. 4.

The reconstruction result experimented on Fountain shows that outline of the statue and the wall reconstructed by the proposed method is more complete than the other methods [5, 10].

Fig. 5.

The reconstruction result experimented on Castle-entry shows that the outline of the castle is clearer, and pillars is more complete than the other methods [5, 10].

For the quantitative evaluation, we compared the reconstructed surface model and GDMR [19] with the real three-dimensional model respectively. We adopted the four standard datasets [18] as the assessment model. Table 1 and Table 2 show the error statistics of the surface model reconstructed by the algorithm and the ground truth on the Fountain and Herzjesu, and the error statistics of the GDMR and ground truth. In Table 1, the accuracy of the model after optimization of the deformation model in this paper is reduced when the error is 0.1. In Table 2, When the error is 0.045 and 0.06, the accuracy of the model is reduced, which is because the partial occlusion area increases the number of vertices and triangles of the occlusion area by triangulation during the deformation optimization process, resulting in an increase in the number of error points and optimization. In addition, from Table 1 and Table 2, it can be seen that the accuracy of the algorithm in this paper is better than that of GDMR. The initial model, deformation optimization model of our method and GDMR are quantitatively analyzed by experiments. The experiments show that our algorithm achieves higher accuracy than GDMR [19].

Fig. 6.

The reconstruction result experimented on Castle shows that the completeness and detail of our approach is reconstructed better than the other methods [5, 10].

Table 1. Relative errors between reconstructed results of our approach, GDMR [19] and the Fountain ground truth.

Table 2. Relative errors between reconstructed results of our approach, GDMR [19] and the Herzjesu ground truth.

6 Conclusion

In this paper, we presented an approach of 3D surface reconstruction based on Delaunay triangulation. The initial surface model is derived by Delaunay triangulation and graph-cut optimization. Then we optimized the initial surface by deformation model and gradient descent to obtain the final surface. We experimented on four standard datasets of Castle-Entry, Castle, Fountain and Herzjesu. The proposed approach could reconstruct high quality three-dimensional surface models of object from the point cloud models with lots of noise and complex topological structures.