LRA: Local Rigid Averaging of Stretchable Non-rigid Shapes

Abstract

We present a novel algorithm for generating the mean structure of non-rigid stretchable shapes. Following an alignment process, which supports local affine deformations, we translate the search of the mean shape into a diagonalization problem where the structure is hidden within the kernel of a matrix. This is the first step required in many practical applications, where one needs to model bendable and stretchable shapes from multiple observations.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

References

  1. Aflalo, Y., Kimmel, R., & Raviv, D. (2013). Scale invariant geometry for nonrigid shapes. SIAM Journal on Imaging Sciences, 6, 1579–1597.

    MathSciNet  Article  MATH  Google Scholar 

  2. Alfakih, A. Y. (2001). On rigidity and realizability of weighted graphs. Linear Algebra and its Applications, 325, 57–70.

    MathSciNet  Article  MATH  Google Scholar 

  3. Alfakih, A. Y., & Wolkowicz, H. (2002). Two theorems on Euclidean distance matrices and Gale transform. Linear Algebra and its Applications, 340, 149–154.

    MathSciNet  Article  MATH  Google Scholar 

  4. Allen, B., Curless, B., & Popović, Z. (2003). The space of human body shapes: Reconstruction and parameterization from range scans. ACM Transactions on Graphics, 22, 587–594.

    Article  Google Scholar 

  5. Amar, A., Wang, Y., & Leus, G. (2010). Extending the classical multidimensional scaling algorithm given partial pairwise distance measurements. IEEE Signal Processing Letters, 17, 473–476.

    Article  Google Scholar 

  6. Avinash, S., Horaud, R., & Diana, M. (2012). 3D shape registration using spectral graph embedding and probabilistic matching. In O. Lezoray & L. Grady (Eds.), Image processing and analysis with graphs: Theory and practice. CRC Press.

  7. Bauer, M., Bruveris, M., & Michor, P. W. (2014). Overview of the geometries of shape spaces and diffeomorphism groups. Journal of Mathematical Imaging and Vision (JMIV), 50, 60–97.

    MathSciNet  Article  MATH  Google Scholar 

  8. Beg, M. F., Miller, M. I., & Younges, T. A. L. (2005). Computing large deformation metric mappings via geodesic flows of diffeomorphisms. International Journal of Computer Vision (IJCV), 61, 139–157.

    Article  Google Scholar 

  9. Belkin, M., & Niyogi, P. (2003). Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15, 1373–1396.

    Article  MATH  Google Scholar 

  10. Ben-Chen, M., Weber, O., & Gotsman, C. (2009). Variational harmonic maps for space deformation. ACM Transactions on Graphics, 28(3). doi:10.1145/1531326.1531340.

  11. Bérard, P., Besson, G., & Gallot, S. (1994). Embedding Riemannian manifolds by their heat kernel. Geometric and Functional Analysis, 4, 373–398.

    MathSciNet  Article  MATH  Google Scholar 

  12. Berger, B., Kleinberg, J., & Leighton, T. (1999). Reconstructing a three-dimensional model with arbitrary errors. Journal of the ACM (JACM), 46, 212–235.

    MathSciNet  Article  MATH  Google Scholar 

  13. Blaschke, W. (1923). Vorlesungen uber Differentialgeometrie und geometrische Grundlagen von Einsteins Relativitatstheorie (Vol. 2). Berlin: Springer.

    Book  MATH  Google Scholar 

  14. Boscaini, D., Eynard, D., Kourounis, D., & Bronstein, M. (2015). Shape-from-operator: Recovering shapes from intrinsic operators. Computer Graphics Forum, 34(2), 265–274.

    Article  Google Scholar 

  15. Bronstein, A. M., Bronstein, M. M., & Kimmel, R. (2006a). Efficient computation of isometry-invariant distances between surfaces. SIAM Journal on Scientific Computing, 28, 1812–1836.

    MathSciNet  Article  MATH  Google Scholar 

  16. Bronstein, A. M., Bronstein, M. M., & Kimmel, R. (2006b). Generalized multidimensional scaling: A framework for isometry-invariant partial surface matching. In Proceedings of National Academy of Science (PNAS) (Vol. 103, pp. 1168–1172).

  17. Chazal, F., Cohen-Steiner, D., Guibas, L., Mémoli, F., & Oudot, S. (2009). Gromov-Hausdorff stable signatures for shapes using persistence. Computer Graphics Forum, 28, 1393–1403.

    Article  Google Scholar 

  18. Coifman, R. R., Lafon, S., Lee, A. B., Maggioni, M., Nadler, B., Warner, F., et al. (2005). Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps. PNAS, 102, 7426–7431.

    Article  Google Scholar 

  19. Cox, T. F., & Cox, M. A. A. (1994). Multidimensional scaling. London: Chapman and Hall.

    MATH  Google Scholar 

  20. Davies, R., Twining, C., Cootes, T., Waterton, J., & Taylor, C. (2002). A minimum description length approach to a minimum description length approach to statistical shape modeling. IEEE Transactions on Medical Imaging, 21, 525–537.

    Article  MATH  Google Scholar 

  21. Devir, Y., Rosman, G., Bronstein, A. M., Bronstein, M. M., & Kimmel, R. (2009). On reconstruction of non-rigid shapes with intrinsic regularization. In Proceedings of workshop on non-rigid shape analysis and deformable image alignment (NORDIA).

  22. Durrleman, S., Pennec, X., Trouvé, A., Braga, J., Gerig, G., & Ayache, N. (2013). Toward a comprehensive framework for the spatiotemporal statistical analysis of longitudinal shape data. PNAS, 103, 22–59.

    MathSciNet  MATH  Google Scholar 

  23. Elad, A.., & Kimmel, R. (2001) Bending invariant representations for surfaces. In Proceedings of computer vision and pattern recognition (CVPR) (pp. 168–174).

  24. Fletcher, P. T., Joshi, S., Lu, C., & Pizer, S. (2003) Gaussian distributions on Lie groups and their application to statistical shape analysis. In Proceedings of information processing in medical imaging (IPMI) (pp. 450–462).

  25. Grimes, C., & Donoho, D. L. (2003). Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data. PNAS, 100, 5591–5596.

    MathSciNet  Article  MATH  Google Scholar 

  26. Hamza, A., & Krim, H. (2006). Geodesic matching of triangulated surfaces. PNAS, 15, 2249–2258.

    Google Scholar 

  27. Heeren, B., Rumpf, M., Schröder, P., Wardetzky, W., & Wirth, B. (2014). Exploring the geometry of the space of shells. PNAS, 33, 247–256.

    Google Scholar 

  28. Hendrickson, B. (1992). Conditions for unique graph realizations. PNAS, 21, 65–84.

    MathSciNet  MATH  Google Scholar 

  29. Hendrickson, B. (1995). The molecule problem: Exploiting structure in global optimization. SIAM Journal on Optimization, 5, 835–857.

    MathSciNet  Article  MATH  Google Scholar 

  30. Huang, H., Shen, L., Zhang, R., Makedon, F., Hettleman, B., & Pearlman, J. D. (2005). Surface alignment of 3D spherical harmonic models: Application to cardiac MRI analysis. In Proceedings of medical image computing and computer assisted intervention (MICCAI) (pp. 67–74).

  31. Ji, X., & Zha, H. (2004). Sensor positioning in wireless ad-hoc sensor networks using multidimensional scaling. In INFOCOM 2004. Twenty-third annual joint conference of the IEEE computer and communications societies (pp. 2652-2661), ed.

  32. Kircher, S., & Garland, M. (2008). Free-form motion processing. ACM Transactions on Graphics, 27, 12.

    Article  Google Scholar 

  33. Kovnatsky, A., Bronstein, M. M., Bronstein, A. M., Glashoff, K., & Kimmel, R. (2013a). Coupled quasi-harmonic bases. PNAS, 32, 439–448.

  34. Kovnatsky, A., Raviv, D., Bronstein, M. M., Bronstein, M. A., & Kimmel, R. (2013b). Geometric and photometric data fusion in non-rigid shape analysis. Numerical Mathematics: Theory, Methods and Applications, 6(1), 199–222.

  35. Li, H., Sumner, R. W., & Pauly, M. (2008). Global correspondence optimization for non-rigid registration of depth scans. Computer Graphics Forum, 27(5), 1421–1430.

    Article  Google Scholar 

  36. Li, R., Turaga, P., Srivastava, A., & Chellappa, R. (2014). Differential geometric representations and algorithms for some pattern recognition and computer vision problems. PNAS, 43, 3–16.

    Google Scholar 

  37. Lipman, Y., & Funkhouser, T. (2009). Möbius voting for surface correspondence. In Proceedings of ACM transactions on graphics (SIGGRAPH) (Vol. 28).

  38. Mémoli, F., & Sapiro, G. (2005). A theoretical and computational framework for isometry invariant recognition of point cloud data. PNAS, 5, 313–347.

    MathSciNet  MATH  Google Scholar 

  39. Miller, M. I., Younes, L., & Trouvé, A. (2014). Diffeomorphometry and geodesic positioning systems for human anatomy. Technology (Singapore World Science), 2, 36–43.

    Google Scholar 

  40. Ovsjanikov, M., Mérigot, Q., Mémoli, F., & Guibas, L. J. (2010). One point isometric matching with the heat kernel. In Proceedings of symposium on geometry processing (SGP) (Vol. 29, pp. 1555–1564).

  41. Patel, A., & Smith, W. A. (2015). Manifold-based constraints for operations in face space. PNAS, 52, 206–217.

    Google Scholar 

  42. Pennec, X. (2006). Intrinsic statistics on Riemannian manifolds: Basic tools for geometric measurements. PNAS, 25, 127–154.

    MathSciNet  Google Scholar 

  43. Praun, E., Sweldens, W., & Schröder, P. (2001). Consistent mesh parameterizations. In Proceedings of ACM transactions on graphics (SIGGRAPH) (pp. 179–184).

  44. Raviv, D., Bronstein, M. M., Bronstein, M. A., & Kimmel, R. (2010a). Full and partial symmetries of non-rigid shapes. International Journal of Computer Vision (IJCV), 89(1), 18–39.

  45. Raviv, D., Bronstein, M. M., Bronstein, M. A., Kimmel, R., & Saprio, G. (2010b). Diffusion symmetries of non-rigid shapes. In Proceedings of 3DPVT.

  46. Raviv, D., Bronstein, A. M., Bronstein, M. M., Waisman, D., Sochen, N., & Kimmel, R. (2014). Equi-affine invariant geometry for shape analysis. Journal of Mathematical Imaging and Vision (JMIV), 50, 144–163.

    MathSciNet  Article  MATH  Google Scholar 

  47. Raviv, D., & Kimmel, R. (2015). Affine invariant non-rigid shape analysis. International Journal of Computer Vision (IJCV), 111, 1–11.

  48. Raviv, D., & Raskar, R. (2015). Scale invariant metrics of volumetric datasets. SIAM Journal on Imaging Sciences, 8, 403–425.

    MathSciNet  Article  MATH  Google Scholar 

  49. Reuter, M., Rosas, H., & Fischl, B. (2010). Highly accurate inverse consistent registration: A robust approach. Neuroimage, 53, 1181–1196.

  50. Rosman, G., Bronstein, M. M., Bronstein, A. M., & Kimmel, R. (2010). Nonlinear dimensionality reduction by topologically constrained isometric embedding. International Journal of Computer Vision (IJCV), 89(1), 56–58.

  51. Roweis, S. T., & Saul, L. K. (2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290, 2323–2326.

  52. Rustamov, R. (2007). Laplace–Beltrami eigenfunctions for deformation invariant shape representation. In Proceedings of symposium on geometry processing (SGP) (pp. 225–233).

  53. Rustamov, R. M., Ovsjanikov, M., Azencot, O., Ben-Chen, M., Chazal, F., & Guibas, L. (2013). Map-based exploration of intrinsic shape differences and variability. ACM Transactions on Graphics, 32, 72.

    Article  MATH  Google Scholar 

  54. Sheffer, A., & Kraevoy, V. (2004). Pyramid coordinates for morphing and deformation. In Proceedings of the 3D data processing, visualization, and transmission (3DPVT).

  55. Shtern, A., & Kimmel, R. (2014). Iterative closest spectral kernel maps. In Proceedings of international conference on 3D vision (3DV).

  56. Silva, V. D., & Tenenbaum, J. B. (2003). Global versus local methods in nonlinear dimensionality reduction. Advances in Neural Information Processing Systems, 15, 705–712.

    Google Scholar 

  57. Singer, A. (2008). A remark on global positioning from local distances. Proceedings of National Academy of Science (PNAS), 105, 9507–9511.

    MathSciNet  Article  MATH  Google Scholar 

  58. Su, B. (1983). Affine differential geometry. Beijing: Science Press.

    MATH  Google Scholar 

  59. Su, J., Kurtek, S., Klassen, E., & Srivastava, A. (2014). Statistical analysis of trajectories on riemannian manifolds: Bird migration, hurricane tracking and video surveillance. The Annals of Applied Statistics, 8, 530–552.

    MathSciNet  Article  MATH  Google Scholar 

  60. Sumner, R. W., & Popović, J. (2004). Deformation transfer for triangle meshes. ACM Transactions on Graphics, 23, 399–405.

    Article  Google Scholar 

  61. Tenenbaum, J., de Silva, V., & Langford, J. (2000). A global geometric framework for nonlinear dimensionality reduction. Science, 290, 2319–2323.

    Article  Google Scholar 

  62. Vaillant, M., Miller, M. I., Younes, L., & Trouve, A. (2004). Statistics on diffeomorphisms via tangent space representations. Neuroimage, 23, 161–169.

    Article  Google Scholar 

  63. Wang, Y., Gupta, M., Zhang, S., Wang, S., Gu, X., Samaras, D., et al. (2008). High resolution tracking of non-rigid motion of densely sampled 3D data using harmonic maps. International Journal of Computer Vision (IJCV), 76, 283–300.

    Article  Google Scholar 

  64. Weber, O., Poranne, R., & Gotsman, C. (2012). Biharmonic coordinates. Computer Graphics Forum, 31(8), 2409–2422.

    Article  Google Scholar 

  65. Winkler, T., Drieseberg, J., Alexa, M., & Hormann, K. (2010). Multi-scale geometry interpolation. Computer Graphics Forum, 29(2), 309–318.

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Dan Raviv.

Additional information

Communicated by Joachim Weickert.

Appendices

Appendix 1: Proof of Lemma 2

A useful Lemma for matrix derivation states that

\(\nabla _A {\mathrm {Tr}} ABA^TC = CAB+C^TAB^T\).

Proof

We denote \(f(A)=AB\), from which it follows that

$$\begin{aligned} \nabla _A {{\mathrm {Tr}}} ABA^TC= & {} \nabla _A {\mathrm {Tr}} f(A)A^TC \nonumber \\= & {} \nabla _{\tilde{A}} {\mathrm {Tr}} f(\tilde{A}) A^TC + \nabla _{\tilde{A}} {\mathrm {Tr}} f(A)\tilde{A} ^T C \nonumber \\= & {} (A^TC)^Tf'(\tilde{A}) + ( \nabla _{\tilde{A}} {\mathrm {Tr}} f(A)\tilde{A} ^T C )^T \nonumber \\= & {} C^T A B^T + (\nabla _ {\tilde{A} ^T} {\mathrm {Tr}} \tilde{A} ^T C f(A))^T \nonumber \\= & {} C^TAB^T + (Cf(A)^T)^T \nonumber \\= & {} C^TAB^T + CAB. \end{aligned}$$
(19)

\(\square \)

Appendix 2: Proof of Corollary 1

$$\begin{aligned} || P^T \varLambda P ||_F^2= & {} {\mathrm {Tr}} (P^T \varLambda P)(P^T \varLambda P)^T \nonumber \\= & {} {\mathrm {Tr}} ( P^T \varLambda P P^T \varLambda P ) \nonumber \\= & {} {\mathrm {Tr}} ( PP^T \varLambda (P^T P )^T \varLambda ). \end{aligned}$$
(20)

Using Lemma 2 and marking \(PP^T=A\) and \(B=C=\varLambda \) we see that

$$\begin{aligned} \nabla _{PP^T} || P^T \varLambda ^T P ||_F^2 = 2\varLambda PP^T \varLambda . \end{aligned}$$
(21)

Because \(\nabla _ P PP^T = 2P\), we use the chain rule and infer that

$$\begin{aligned} \nabla _{P} || P^T \varLambda ^T P ||_F^2= & {} 2\varLambda PP^T \varLambda \nabla _P PP^T \nonumber \\= & {} 4\varLambda PP^T \varLambda P. \end{aligned}$$
(22)

Appendix 3: Proof of Corollary 2

Since

$$\begin{aligned} V_{ij}PP^T V_{ij}^T= & {} {\mathrm {Tr}} (V_{ij}PP^T V_{ij}^T) \\= & {} {\mathrm {Tr}} (PP^T V_{ij}^T V_{ij}), \end{aligned}$$

we can once again use Lemma 2. We denote \(P=A\), B = \({\mathcal {I}}\), and C = \(V_{ij}^T V_{ij}\), and infer that

$$\begin{aligned} \nabla _P V_{ij}PP^T V_{ij}^T = 2V_{ij}V_{ij}^T P. \end{aligned}$$
(23)

From the chain rule we readily have that

$$\begin{aligned}&\nabla _P \left( V_{ij}PP^T V_{ij}^T - d_{ij} \right) ^2 \nonumber \\&\quad =2 \left( V_{ij}PP^T V_{ij}^T - d_{ij}^2 \right) \nabla _P V_{ij}PP^T V_{ij}^T \nonumber \\&\quad = 4 \left( V_{ij}PP^T V_{ij}^T - d_{ij}^2 \right) V_{ij}V_{ij}^T P. \end{aligned}$$
(24)

Appendix 4: Aligning Stretchable Structures

We present a new alignment algorithm which is a merger of a traditional non-rigid alignment approach and a global intrinsic regularization which compensates for stretching. During the iterations we search for neighbors only for points which share similar intrinsic (long) distances from known anchor points. Since those distances were generated using a local metric which is invariant for stretching, this regularization is also invariant to such deformations, without the need to know a priori their location or strength. One should think about the non-rigid ICP procedure as a projection operator from space into the structure, while the intrinsic constraints are responsible for achieving an intrinsic alignment. In what follows we show how to generate long geodesics which are locally invariant to stretching. We first summarize known results on diffusion geometry, where given the Laplace Beltrami operator one can define long distances. Next we show how to build a local metric which is invariant to affine deformations and is used to empower the Laplace Beltrami operator with stretching invariant properties. This appendix can not replace the cited papers, but provides a descent summary to understand the steps required for alignment.

Spectral geometry: from local metric to global distances:

We model a shape as a complete two dimensional Riemannian manifold X with a metric tensor g, noted by (Xg). As we assume X is embedded into \({\mathbb {R}}^3\) by a regular map \(\mathbf {x} : U \subseteq {\mathbb {R}}^2 \rightarrow {\mathbb {R}}^3\), the metric tensor can be expressed in coordinates as the coefficients of the first fundamental form \( g_{ij} = \langle \frac{\partial \mathbf {x}}{\partial u_i} , \frac{\partial \mathbf {x} }{\partial u_j}\rangle \), where \(u_i\) are the coordinates of U. Hence, the arc-length becomes \(dp^2 = g_{11}{du_1}^2 + 2g_{12}du_1 du_2 + g_{22}{du_2}^2\).

Spectral geometry is based on the partial differential equation

$$\begin{aligned} \left( \frac{\partial }{\partial t} + \Delta _g \right) f(t,x) = 0, \end{aligned}$$
(25)

which is called the heat equation. It describes heat propagation, where f(tx) is the heat distribution at a point x in time t , and \(\Delta _g\) is the Laplace-Beltrami operator (LBO) which is evaluated from the local metric g. The fundamental solution of the heat equation is called the heat kernel and can be expressed using the spectral decomposition of \(\varDelta _g\),

$$\begin{aligned} h_t(x,x') = \sum _{i\ge 0} e^{-\lambda _i t} \phi _i(x) \phi _i(x'), \end{aligned}$$
(26)

where \(\phi _i\) and \(\lambda _i\) are, respectively, the eigenfunctions and eigenvalues the \(\varDelta _g\). Out of the heat kernel we can construct a family of intrinsic metrics known as diffusion metrics,

$$\begin{aligned} d^2_t(x,x')= & {} \int \left( h_t(x,\cdot ) - h_t(x',\cdot ) \right) ^2 da \nonumber \\= & {} \sum _{i> 0} e^{-2\lambda _i t} (\phi _i(x) - \phi _i(x'))^2, \end{aligned}$$
(27)

which measure the diffusion distance of the two points for a given time t. The parameter t can be considered as scale, and the family \(\{ d_t \}\) can be thought of as a scale-space of metrics. Further details for numerically constructing a metric dependent Laplace-Beltrami operator can be found in Raviv et al. (2014) and Raviv and Kimmel (2015).

Locally affine metric invariants:

The seminal work of Blaschke (1923) and Su (1983) showed that if scaling is known, an equi-affine metric, also known as special-affine, can be constructed on a two dimensional (surface) manifold, where the equi-affine re-parametrization invariant metric (Su 1983; Blaschke 1923) reads \(q_{ij} = |r|^{-\frac{1}{4}}r_{ij}\), where \(r_{ij} = \det \left( \frac{\partial \mathbf {x}}{\partial u_1} , \frac{\partial \mathbf {x}}{\partial u_2} , \frac{\partial ^2 \mathbf {x}}{\partial u_1 \partial u_2} \right) \), and \(r = \det (r_{ij}) = r_{11}r_{22}-r_{12}^2\). This scheme and its applications was recently examined in Raviv et al. (2014), where we turned this local pseudo metric into a practical tool.

In a parallel effort we studied scale invariant metrics using curvature for regularization (Aflalo et al. 2013). Specifically, consider the surface (Xg), then the scale invariant metric takes the form \( \left| K \right| g_{ij}\), where K is the Gaussian curvature. We further showed how to normalize higher order non-rigid structures (Raviv and Raskar 2015), and showed its usefulness in medical volumetric data (e.g., Computed Tomography).

Surprisingly, it is possible to combine the two results, equi-affine and scaling, in one framework and removing the annoying equi limitation. First shown in Raviv and Kimmel (2015), one needs to replace the Gaussian curvature in the scale invariant metric definition with an equi-affine invariant curvature \(K^q\), leading to \(a_{ij} = \left| K^q \right| q_{ij}\), a locally affine invariant metric. Going back to diffusion distances, we need to replace the metric \(g_{ij}\) with the metric \(a_{ij}\) in order to evaluate affine invariant distances. Notice that the invariance is a local property leading to a global invariance in distance measures. One does not need to know a priori where and by how much stretching occurred. More details on the numeric construction can be found in Raviv and Kimmel (2015).

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Raviv, D., Bayro-Corrochano, E. & Raskar, R. LRA: Local Rigid Averaging of Stretchable Non-rigid Shapes. Int J Comput Vis 124, 132–144 (2017). https://doi.org/10.1007/s11263-017-1002-1

Download citation

Keywords

  • Non-rigid shapes
  • Metric invariants
  • Affine invariant
  • Averaging shapes
  • Shape space
  • Non-rigid statistics