# A Symmetry Prior for Convex Variational 3D Reconstruction

- 9 Citations
- 11k Downloads

## Abstract

We propose a novel prior for variational 3D reconstruction that favors symmetric solutions when dealing with noisy or incomplete data. We detect symmetries from incomplete data while explicitly handling unexplored areas to allow for plausible scene completions. The set of detected symmetries is then enforced on their respective support domain within a variational reconstruction framework. This formulation also handles multiple symmetries sharing the same support. The proposed approach is able to denoise and complete surface geometry and even hallucinate large scene parts. We demonstrate in several experiments the benefit of harnessing symmetries when regularizing a surface.

## Keywords

Symmetry prior 3D reconstruction Variational methods Convex optimization## 1 Introduction

### 1.1 Contributions

We propose to use symmetry information as a prior in 3D reconstruction in order to favor symmetric solutions when dealing with noisy and incomplete data. For this purpose, we extend standard symmetry detection algorithms to be able to exploit partially unexplored domains. Our framework naturally unifies the applications of symmetry-based surface denoising, completion and the hallucination of unexplored surface areas. To the best of our knowledge, we present the first method that handles multiple symmetries with a shared support region, since the proposed algorithm computes an approximation to a non-trivial projection to equally satisfy a set of symmetries. Finally, our method extends the toolbox of priors for many existing variational 3D reconstruction methods and we show that a symmetry prior can achieve quality improvements with a moderate runtime overhead.

### 1.2 Related Work

The works by Liu *et al.* [13] and Mitra *et al.* [16] give a broad overview of symmetry detection methods, although most of the work they discuss focuses on symmetry extraction for computer graphics applications. Therefore, their applicability is mostly shown on perfect synthetic data. In this section, we focus on works that detect and exploit symmetries from real data for vision applications.

Köser *et al.* [11] detect planar reflective symmetries in a single 2D image. They demonstrate that the arising stereo problem can be solved with standard stereo matching when the distance of the camera to the symmetry plane corresponds to a reasonable baseline.

*et al.*[7] define a reflective symmetry descriptor for 3D models by continuously measuring symmetry scores for all planes through the models’ center of mass. We briefly repeat some of their theoretic results as we are going to use them in Sect. 2. Let \(\gamma \in \varGamma \) be a symmetric transformation and

*u*be a symmetric object, then

*u*is symmetric with respect to \(\gamma \) if it is invariant under the symmetric transformation, that is, \(\gamma (u)=u\). Using the group properties of the symmetry transform, Kazhdan

*et al.*define the following symmetry distance

*u*onto the set of objects which are symmetric with respect to the symmetry \(\gamma \).

Based on these results, Podolak *et al.* [23] focused on planar reflective symmetries and generalized the descriptor to detect also non-object-centered symmetries. They further define geometric properties which can be used for model alignment or classification. Both [4, 22] detect symmetries in meshes and propose a symmetric remeshing of objects for quality improvements of the mesh and for more consistent mesh approximations during simplification operations. Similarly, Mitra *et al.* [15] use approximate symmetry information of objects to allow their transformation into perfectly symmetric objects.

Cohen *et al.* [2] detect symmetries in sparse point clouds by using appearance-based 3D-3D point matching in a RANSAC-based procedure. They then exploit these symmetries to remove drift from point-clouds. Although their work is done on sparse point clouds, we present a similar approach for voxel grids in Sect. 2.

Symmetries have also been used for many applications, e.g. shape matching and feature point matching [6], object part segmentation, and canonical coordinate frame selection [23]. Our goal is to apply these concepts into the domain of dense 3D reconstruction. Nevertheless, our work is not the first one leveraging symmetry information in this domain. One of the earliest attempts to incorporate symmetry priors into surface reconstruction methods was by Terzopoulos *et al.* [25] who use a deformable spine model to create a generalized cylinder shape from a single image.

Application-wise, the work by Thrun and Wegbreit [26] is closely related to ours. They detect an entire hierarchy of different symmetry types in point cloud data and subsequently demonstrate the completion of unexplored surface parts. As opposed to our approach, they do not simultaneously denoise the input data and they do not compute a water-tight surface.

In contrast, we propose to integrate knowledge about symmetries directly into the surface reconstruction process in order to better reason about noisy or incomplete input data which can come from image-based matching algorithms or 3D depth sensors. Furthermore, our approach can handle any number of symmetries and their support domains can arbitrarily overlap. To the best of our knowledge no other method deals with several symmetries that share the same domain in a way that the “symmetrized” result obeys the group structure of several symmetries at once.

We build our symmetry prior into the variational volumetric 3D reconstruction framework which has been used in various settings, e.g. depth map fusion [29], 3D reconstruction [9, 27], multi-label semantic reconstruction [5], and spatio-temporal reconstruction [17], with anisotropic regularization [5, 10, 18] or connectivity constraints [19]. The proposed method extends these lines of works by a symmetry prior which can be easily combined with any of these works. Moreover, since the 3D reconstruction is formulated as a 3D segmentation problem, the proposed prior is also directly applicable to a large set of segmentation methods like [24, 28].

## 2 Symmetry Detection

In order to exploit symmetry priors for reconstruction, we first need to detect the symmetries that best fit the data. In this paper, we focus on detecting planar symmetries. As input we use integrated depth information, which is represented by a truncated signed distance function (TSDF) on a volume \(V\subset \mathbb {R}\) [3]. The TSDF assigns a positive value for voxels corresponding to free space and a negative value for occupied voxels (which are placed behind the observed depth values). A zero value in this function denotes an unobserved (or occluded) voxel. A surface is then implicitly defined as the transition between positive and negative values. Since we are only interested in voxels lying on the surface of the object, we only look at the voxels for which the gradient is very strong.

Furthermore, since planar surfaces are symmetric with respect to all planes perpendicular to them, which is not very informative for the global scene or even for a small object, we decide to look only at those voxels that exhibit a certain degree of curvature. Thus, the goal is to find the symmetry planes that best reflect these high-gradient and high-curvature voxels into positions that also have a large gradient and curvature or that are otherwise unknown or occluded. Note that [8] detect partial symmetries in volumetric data via sparse matching of extracted line features, which is a more sophisticated way of using the gradient on the data. While trying to find the symmetries of a scene, we define the symmetry support \(V_\gamma \subseteq V\subset \mathbb {R}^3\) as a hole-free, connected subset of the reconstruction domain \(V\) that fits the detected symmetry, that is, we try to include all occupied and free space which complies with the symmetry \(\gamma \). Unobserved regions will be included an treated in a way such that they perfectly fulfill the symmetry to allow for hole filling and hallucination.

### 2.1 RANSAC

We apply a RANSAC-based approach by taking as input the list of high-gradient and high-curvature voxels, which we will refer to as surface voxels, and the list of unknown voxels. First, we randomly sample two surface voxels, which define a unique symmetry plane that reflect one of these voxels into the other. Next, we look through all of the surface voxels and reflect them over this plane to look for inliers to this particular symmetry. If the reflection of a surface voxel falls into the position of another surface voxel, then they are both considered as inliers to this symmetry plane. However, if it falls into an unknown voxel position, then we also consider it as an inlier since this could be a potential occluded part of a symmetric object. We randomly sample planes from two surface voxels as many times as stated by the RANSAC termination formula.

The plane with the most inliers is chosen as the best global symmetry plane \(\gamma \) for this surface, and its inliers define the support \(V_\gamma \). Since we are interested in potentially finding many symmetries for the same scene, RANSAC could be applied sequentially by removing the inliers for the best symmetry and then subsequently re-detecting the next best symmetry among the remaining surface voxels. However, since there could be many symmetries with the same support, we modify RANSAC and keep track of the set of best *N* solutions (where *N* has to be determined a-priori). This way, we extract the *N* symmetries that best fit the entire surface. We will refer to this set of symmetries as \(\varGamma \).

*Local Symmetry Detection.* A scene can be composed of several objects with different sizes and symmetries. However, due to the size variability, applying RANSAC on the whole volume as described before would miss most of the symmetries for small objects and also cluster different objects with similar symmetry planes into one approximate, noisy, symmetry plane. Therefore, we apply the described RANSAC approach for sliding boxes of different sizes over the entire volume. This allows the segmentation of objects as the support for the different symmetries found. The symmetry planes with the bigger inlier ratios and their support are chosen as candidates for local object symmetries. For multiple detections of the same object (parts) at different scales we gave preference to larger support domains and rejected detections whose support is a subset of another.

### 2.2 Hough Transform

If we consider the special case when *u* values are binaries, representing an occupancy grid, the Eq. (2) becomes the number of inliers of the \(\gamma \) symmetric plane, a metric also used in the RANSAC method described previously. Therefore, the methods are essentially very similar and the decision between one or the other depends only on technical considerations. For example, the runtime of Hough Transform is fixed, which is the number of iterations used in the plane sampling; on the other hand, the number of iterations ran by RANSAC depends on the current inlier threshold and, therefore, could possibly finish sooner. Another advantage of RANSAC is its low memory footprint and the fact that it doesn’t require a non-maximal suppression step. However, knowing the Hough space can be a handy tool to visually understand the type of symmetries obtained in the process.

One should notice that there are more sophisticated methods in the literature to better extract symmetries, for instance [15, 26]. However, the main contribution of the paper is the use of these detected symmetries in a variational optimization framework, described in the following section.

## 3 Surface Reconstruction with a Symmetry Prior

*D*is the derivative in a distributional sense. The last term of Eq. (3) minimizes the distance of all symmetries given by \(\varGamma \) for which we use a slightly modified distance measure (Eq. (1)) in such a way that the distance is only evaluated for points within the corresponding support domain \(V_\gamma \). Furthermore, the weights \(\omega _\gamma \) can be used to change the impact among individual symmetries.

*Symmetry Projection.*For a given set of

*m*symmetries \(\varGamma = \{\gamma _1, \gamma _2, \ldots , \gamma _m\}\), minimizing energy (3) approximates the joint projection onto a set of symmetries

*Remark.*As shown in [7] and stated in Eq. (1), the projection \(\varPi _\gamma (u)\) of function

*u*onto a single symmetry has a simple analytic solution: being the average of

*u*and \(\gamma (u)\). This is not the case for the projection onto a set of symmetries. Figure 3 illustrates that even the projection \(\varPi _{\gamma _1, \gamma _2}(u)\) onto only two symmetries is not a simple combination of the individual projections \(\varPi _{\gamma _1}(u), \varPi _{\gamma _2}(u)\). Since our method minimizes the symmetry distance to an arbitrary set of symmetries, the minimizer of energy (3) for an infinite symmetry term weight (\(\mu \rightarrow \infty \)) approaches the joint projection \(\varPi _\varGamma \) onto all symmetries in \(\varGamma \) which inherently generates a complex symmetry group as a combination of the input symmetries.

## 4 Numerical Optimization

*u*is a stacked vector of all voxels in the volume domain. The symmetry dependencies between different points can be represented as a linear transformation, that is, the symmetry distance in Eq. (3) can be rewritten as \(\text {SD}^2(u, \gamma ) = \left\| {A_\gamma u-b}\right\| ^2\). As a result, all three terms of the functional are convex and we can efficiently minimize energy (3) with the preconditioned first-order primal-dual algorithm [21]. In our setting the algorithm alternatingly iterates the following projected gradient ascent, gradient decent and linear extrapolation steps.

The primal-dual surface optimization lends itself to a parallel implementation for which we used the CUDA framework. Without further processing, we finally extract a mesh as the 0.5 iso-surface from the implicit surface representation *u* using the Marching cubes [14] algorithm.

Although the most common types of symmetries, like planar reflective symmetries, rotational, and translational symmetries are represented by linear transformations, the convexity of the energy is also not violated if the symmetry transformation is non-linear. This is because the transformation affects only the argument *x* of *u*, but not *u* itself. Hence, our formulation can handle any type of symmetry. In this work, we focus, however, on planar reflective symmetries since these are the most common ones in many environments.

*Planar Reflective Symmetries.*In this case the symmetry \(\gamma \) is parametrized by a 3D plane given in Hessian normal form by \(nx-d=0\) with unit normal \(n\in \mathbb {R}^3\) and the distance from the origin \(d \in \mathbb {R}\). The linear transformation inside the symmetry term in Eq. (5) is then given by \(b=0\), \(A_\gamma = (\mathbb {I}-M_\gamma )\), with \(\mathbb {I}\) being the identity matrix. The binary matrix \(M_\gamma \in \{0,1\}^{\left| {V}\right| \times \left| {V}\right| }\) (with \(\left| {V}\right| \) being the number of voxels in the discretized volume domain) encodes all pairwise dependencies between voxels being linked by the symmetry \(\gamma \) and is defined as

## 5 Experiments

Although our approach inherently combines surface denoising, completion and hallucination, we try to isolate and evaluate these properties separately in the following experiments.

### 5.1 Surface Denoising and Completion

### 5.2 Surface Hallucination

### 5.3 Local Symmetries

Figure 6 shows an experiment with multiple local symmetries of objects in a desk scene. The local symmetries were detected using the RANSAC approach with sliding search boxes as described in Sect. 2.1. The first row in Fig. 6 depicts the input data which was scanned with a structured light RGB-D sensor and is missing surface information for several unobserved scene parts. The baseline reconstruction without the symmetry term (\(\mu =0\)) is shown in the second row of Fig. 6. Some of its areas are filled with minimal surfaces as visible in the backside of the monitor and the lower part of the office chair. In contrast, the results with our approach (third row of Fig. 6) demonstrate that these areas are filled in a more meaningful way with the help of the symmetry information.

### 5.4 Discussion and Limitations

*Parameter Settings.* We experimented with several settings for the individual symmetry weights \(\omega _\gamma \), like e.g. the support size, inlier ratios, or number of inliers, following the idea that symmetries with a stronger data support should also be enforced in a stronger way. However, we found that uniform weights (\(\forall \gamma : \omega _\gamma = 1\)) gave the best results and used them in all of our experiments. The other model parameters (volume resolution, data term weight \(\lambda \), and symmetry term weight \(\mu \)) are summarized in Table 1. The choice of these parameters is intuitive and careful tuning was not necessary. In order to weigh the amount of smoothness, data fidelity and symmetry against each other, we found as a rule of thumb that the data term weight and the symmetry term weight should be changed in a similar manner in order to keep a similar amount of smoothness, e.g. when enforcing more symmetry, the data term weight should be raised as well to maintain the amount of data fidelity. Conversely, for less smoothness the data term weight is raised and the symmetry term weight needs to be raised as well in order to maintain a comparable impact of the symmetry term. For more sophisticated symmetries like the stool or the toy example in Fig. 3, large symmetry term weights are required to enforce the full group structure.

Overview of parameters for all experiments. The first two columns show the voxel resolution next to the corresponding dataset and the last two columns show the data term weight \(\lambda \) and the symmetry term weight \(\mu \) (see Eq. (3)).

Dataset | Voxel resolution \(\left| {V}\right| \) | \(\lambda \) | \(\mu \) |
---|---|---|---|

Stool | \({191}\times {158}\times {135}\) (4M) | \(10^8\) | \(10^7\) |

Capitol | \({700}\times {560}\times {560}\) (220M) | 1.5 | 6 |

g-hall | \({512}\times {512}\times {213}\) (56M) | 0.75 | 6 |

Desk | \({311}\times {289}\times {239}\) (21M) | 1500 | 300 |

Hallway | \({730}\times {436}\times {301}\) (96M) | 200 | 200 |

*Computation Time.* All results have been computed on a Linux-based i7 CPU with a GeForce GTX 780 graphics card. While the symmetry detection is performed by unoptimized CPU code, the surface optimization with a symmetry prior has been implemented on the GPU. The computation times for the symmetry detection vary depending on the dataset and are within seconds for simple scenes like the stool and up to 2 h for the sliding box approach on the desk dataset. The computation time for the surface optimization with and without symmetry constraints with 80M voxels were 2:20 min and 1:03 min, respectively. This highly depends on the grid resolution, e.g., the reconstruction of the stool in Fig. 1 with 4M voxels took only 5 s with- and 2 s without symmetry constraints.

## 6 Conclusion

In this paper we proposed a novel symmetry prior for variational 3D reconstruction. Our method is able to enforce several symmetries with the same support area, allowing for symmetry interactions that proved to be very useful for several applications such as surface denoising and completion, as well as surface hallucination in the case of highly incomplete data. We also discussed two planar symmetry detection approaches and how to handle and exploit unobserved areas in order to more robustly detect such symmetries. We showed the results of our method in several datasets, ranging from noisy and slightly incomplete reconstructions, to models with almost half of its surface missing. We also showed results on scenes with several local symmetries with different support sizes. In future work, we would like to explore better symmetry detection methods, as well as experiment with different kinds of symmetries, such as rotational, translational or curved symmetries (e.g. as in [12]).

## Notes

### Acknowledgments

This work was supported by the Horizon 2020 research and innovation programme under grant agreement No. 637221, and by the CTI Switzerland grant No. 17136.1 Geometric and Semantic Structuring of 3D point clouds.

## References

- 1.Cohen, A., Sattler, T., Pollefeys, M.: Merging the unmatchable: stitching visually disconnected SFM models. In: ICCV, December 2015Google Scholar
- 2.Cohen, A., Zach, C., Sinha, S., Pollefeys, M.: Discovering and exploiting 3D symmetries in structure from motion. In: CVPR, June 2012Google Scholar
- 3.Curless, B., Levoy, M.: A volumetric method for building complex models from range images. In: SIGGRAPH (1996)Google Scholar
- 4.Golovinskiy, A., Podolak, J., Funkhouser, T.: Symmetry-aware mesh processing. In: Hancock, E.R., Martin, R.R., Sabin, M.A. (eds.) Mathematics of Surfaces XIII. LNCS, vol. 5654, pp. 170–188. Springer, Heidelberg (2009)CrossRefGoogle Scholar
- 5.Häne, C., Zach, C., Cohen, A., Angst, R., Pollefeys, M.: Joint 3D scene reconstruction and class segmentation. In: CVPR, pp. 97–104 (2013)Google Scholar
- 6.Hauagge, D.C., Snavely, N.: Image matching using local symmetry features. In: CVPR, pp. 206–213 (2012)Google Scholar
- 7.Kazhdan, M.M., Chazelle, B., Dobkin, D.P., Funkhouser, T.A., Rusinkiewicz, S.: A reflective symmetry descriptor for 3D models. Algorithmica
**38**(1), 201–225 (2003)CrossRefzbMATHMathSciNetGoogle Scholar - 8.Kerber, J., Wand, M., Krüger, J.H., Seidel, H.: Partial symmetry detection in volume data. In: Proceedings of the Vision, Modeling, and Visualization Workshop, Berlin, Germany, 4–6 October 2011, pp. 41–48 (2011)Google Scholar
- 9.Kolev, K., Klodt, M., Brox, T., Cremers, D.: Continuous global optimization in multiview 3D reconstruction. IJCV
**84**(1), 80–96 (2009)CrossRefGoogle Scholar - 10.Kolev, K., Pock, T., Cremers, D.: Anisotropic minimal surfaces integrating photoconsistency and normal information for multiview stereo. In: ECCV, Heraklion, Greece, September 2010Google Scholar
- 11.Köser, K., Zach, C., Pollefeys, M.: Dense 3D reconstruction of symmetric scenes from a single image. In: Mester, R., Felsberg, M. (eds.) DAGM 2011. LNCS, vol. 6835, pp. 266–275. Springer, Heidelberg (2011)CrossRefGoogle Scholar
- 12.Liu, J., Liu, Y.: Curved reflection symmetry detection with self-validation. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010, Part IV. LNCS, vol. 6495, pp. 102–114. Springer, Heidelberg (2011)CrossRefGoogle Scholar
- 13.Liu, Y., Hel-Or, H., Kaplan, C.S., Gool, L.V.: Computational symmetry in computer vision and computer graphics. Found. Trends Comput. Graph. Vis.
**5**(12), 1–195 (2010)zbMATHGoogle Scholar - 14.Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3D surface construction algorithm. SIGGRAPH Comput. Graph.
**21**, 163–169 (1987)CrossRefGoogle Scholar - 15.Mitra, N.J., Guibas, L.J., Pauly, M.: Symmetrization. ACM Trans. Graph.
**26**(3), 63 (2007)CrossRefGoogle Scholar - 16.Mitra, N.J., Pauly, M., Wand, M., Ceylan, D.: Symmetry in 3D geometry: extraction and applications. Comput. Graph. Forum
**32**(6), 1–23 (2013)CrossRefGoogle Scholar - 17.Oswald, M.R., Cremers, D.: A convex relaxation approach to space time multi-view 3D reconstruction. In: ICCV - Workshop on Dynamic Shape Capture and Analysis (4DMOD) (2013)Google Scholar
- 18.Oswald, M.R., Cremers, D.: Surface normal integration for convex space-time multi-view reconstruction. In: Proceedings of the British Machine and Vision Conference (BMVC) (2014)Google Scholar
- 19.Oswald, M.R., Stühmer, J., Cremers, D.: Generalized connectivity constraints for spatio-temporal 3D reconstruction. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part IV. LNCS, vol. 8692, pp. 32–46. Springer, Heidelberg (2014)Google Scholar
- 20.Pizlo, Z., Sawada, T., Li, Y., Kropatsch, W.G., Steinman, R.M.: New approach to the perception of 3D shape based on veridicality, complexity, symmetry and volume. Vision. Res.
**50**, 1–11 (2010)CrossRefGoogle Scholar - 21.Pock, T., Chambolle, A.: Diagonal preconditioning for first order primal-dual algorithms in convex optimization. In: ICCV, Washington, DC, USA, pp. 1762–1769 (2011)Google Scholar
- 22.Podolak, J., Golovinskiy, A., Rusinkiewicz, S.: Symmetry-enhanced remeshing of surfaces. In: Proceedings of the Fifth Eurographics Symposium on Geometry Processing, Barcelona, Spain, 4–6 July 2007, pp. 235–242 (2007)Google Scholar
- 23.Podolak, J., Shilane, P., Golovinskiy, A., Rusinkiewicz, S., Funkhouser, T.A.: A planar-reflective symmetry transform for 3D shapes. ACM Trans. Graph.
**25**(3), 549–559 (2006)CrossRefGoogle Scholar - 24.Reinbacher, C., Pock, T., Bauer, C., Bischof, H.: Variational segmentation of elongated volumetric structures. In: CVPR (2010)Google Scholar
- 25.Terzopoulos, D., Witkin, A., Kass, M.: Symmetry-seeking models and 3D object reconstruction. IJCV
**1**, 211–221 (1987)CrossRefGoogle Scholar - 26.Thrun, S., Wegbreit, B.: Shape from symmetry. In: ICCV, Bejing, China. IEEE (2005)Google Scholar
- 27.Ummenhofer, B., Brox, T.: Dense 3D reconstruction with a hand-held camera. In: Pinz, A., Pock, T., Bischof, H., Leberl, F. (eds.) DAGM and OAGM 2012. LNCS, vol. 7476, pp. 103–112. Springer, Heidelberg (2012)CrossRefGoogle Scholar
- 28.Unger, M., Pock, T., Cremers, D., Bischof, H.: TVSeg - interactive total variation based image segmentation. In: Proceedings of the British Machine and Vision Conference (BMVC), Leeds, UK, September 2008Google Scholar
- 29.Zach, C., Pock, T., Bischof, H.: A globally optimal algorithm for robust TV-l1 range image integration. In: ICCV, pp. 1–8 (2007)Google Scholar