A fast modal space transform for robust nonrigid shape retrieval

Abstract

Nonrigid or deformable 3D objects are common in many application domains. Retrieval of such objects in large databases based on shape similarity is still a challenging problem. In this paper, we take advantages of functional operators as characterizations of shape deformation, and further propose a framework to design novel shape signatures for encoding nonrigid geometries. Our approach constructs a context-aware integral kernel operator on a manifold, then applies modal analysis to map this operator into a low-frequency functional representation, called fast functional transform, and finally computes its spectrum as the shape signature. In a nutshell, our method is fast, isometry-invariant, discriminative, smooth and numerically stable with respect to multiple types of perturbations. Experimental results demonstrate that our new shape signature for nonrigid objects can outperform all methods participating in the nonrigid track of the SHREC’11 contest. It is also the second best performing method in the real human model track of SHREC’14.

Appendices

Appendix A: Error bounds under deformation

In this section, we investigate error bounds of the spectral signature of integral kernel operators in the presence of “smooth” manifold deformations using perturbation theory [14, 18]. The general result, intuitively speaking, is that the total deviation of a spectral signature with an arbitrary number of dimensions is finitely bounded under smooth deformations. Given a general matrix $A$ and a perturbation parameter $\epsilon$ within a neighborhood of zero, the perturbed matrix is written as $A+\epsilon B$, where $B$ is an arbitrary matrix. In this situation, each eigenvalue or eigenvectors of $A+\epsilon B$ admits an expansion in fractional powers of $\epsilon$, and the zeroth order term of this expansion is an eigenvalue or eigenvector of the unperturbed matrix $A$ (Lidskii theorem, 1965 [25]). This is a well-known result in regular perturbation theory. In particular, one can actually derive a Lipschitz condition number for this continuity:

Theorem 6.1

(A special case of Lidskii theorem [18]) Assuming $\mu (\epsilon )$ is an eigenvalue of the perturbed matrix $A+\epsilon B$ for a sufficiently small $\epsilon$, it admits a first-order expansion

$$\begin{aligned} \mu (\epsilon ) = \mu + \left\| \Phi ^T B\Phi \right\| _{\max } \epsilon + o(\epsilon ), \end{aligned}$$

where $\Phi$ is the eigenvector of $A$ corresponding to eigenvalue $\mu$, $A$ and $B$ are conjugated matrices, $\left\| \cdot \right\| _{\max }$ represents the largest singular value. The first-order coefficient $\left\| \Phi ^T B\Phi \right\| _{\max }$ is called the Lipschitz condition number.

For the rest of this section, $A$ is called characterization, and $B$ is called perturbation. The Lipschitz condition number characterizes the total amount of deviation in a signature with respect to a certain amount of perturbation. In the ideal case of isometric perturbation, where $A$ is isometric and $B$ is zero, the Lipschitz condition number is degenerated. Let $M$ and $M'$ be two compact manifold domains, and $H$ and $H'$ be the spaces of bounded and continuous linear functionals defined on $M$ and $M'$. In the current context, we consider $M'$ as a “gently” deformed version of $M$. We assume that implicitly there exists a matching (or registration) between these two domains such that it induces a mapping between $H$ and $H'$, which is a closed and densely defined operator $\mathcal D$ from $H$ to $H'$. The characterization of this mapping is a self-adjoint operator $\mathcal O$. Suppose $\mathcal O'(\epsilon )$ has a first-order approximate perturbed (infinitely dimensional) matrix representation $A + \epsilon B$ with respect to an underlying domain deformation $\mathcal D_{\epsilon }$.

Definition 6.1

(Perturbated $(\alpha ,p)$ -smoothness) Consider a perturbated matrix $A + \epsilon B$ with a spectral family $\{\mu _i(\epsilon )\}$ admitting first-order expansions: $\mu _i(\epsilon ) = \mu _i + \left| \phi _i^T B\phi _i \right| \epsilon + o(\epsilon )$. $A + \epsilon B$ is perturbated $(\alpha ,p)$-smooth if

$$\begin{aligned} \left\| A^{-\frac{\alpha }{2}}B^pA^{-\frac{\alpha }{2}} \right\| _1=: \sum \limits _{i=1}^{\infty }\left| \mu _i \right| ^{-\alpha }\left\| \phi _i^T B\phi _i \right\| ^p < C(\alpha , p), \end{aligned}$$

for some $\alpha \ge 0$ and $p> 0$, where $\left\| \cdot \right\| _1$ is Schatten p-norms.

Perturbed smoothness actually admits a substantially larger class of deformations of practical interest. It allows the characterization to be varied with respect to the domain deformation, but it bounds the amount of deviation.

Theorem 6.2

(Error bound of spectral signature) Consider a discrepancy function for a pair of original and perturbed spectra

$$\begin{aligned} d_n(\mu ,\mu ';\alpha , p) :=\sum _{i=1}^n\left| \mu _i \right| ^{-\alpha }\left| \mu _i-\mu _i' \right| ^p, \end{aligned}$$

and two Hilbert spaces $H$ and $H'$ admitting a perturbation path $\mathcal D_{\epsilon }$ for $\epsilon \in [0,1]$. If the perturbation path is uniformly perturbed $(\alpha , p)$-smooth with a bound $C(\alpha ,p)$ with respect to a characterization operator $\mathcal O(\epsilon )$, and order preserving, aka $\mu _i\ge \mu _j\iff \mu _i'\ge \mu _j'$ for any $i,j$, we have

$$\begin{aligned} d_n(\mu ,\mu ';\alpha , p)\le \kappa ^{-\alpha } C(\alpha , p), \quad \forall n \end{aligned}$$

where $\kappa = \max \limits _{i}\{\left| \frac{\mu _i}{\mu _{i+1}} \right| ,\left| \frac{\mu _{i+1}}{\mu _i} \right| \}$.

Proof

(Sketch) Take the integral w.r.t $\epsilon$: $\mu _i'-\mu _i=\int _0^1 \mathrm {d}\mu _i(\epsilon )$ and use the condition of perturbated $(\alpha ,p)$-smoothness (Definition 6.1). $\square$

Integral kernel operator Experimental evidences suggest that distance maps/matrices, more generally, integral operators, could be a useful source for shape descriptor construction, and are potentially better than differential operators (e.g., LBO). As a rigorous treatment justifying the capability of integral kernel operators, we establish another definition for the smoothness of perturbations:

Definition 6.2

($\alpha$ -Hölder class of perturbed kernel) Consider integral kernel operator $k_{\epsilon }(\cdot ,\cdot )$ with perturbation parameter $\epsilon$, and define the perturbed kernel

$$\begin{aligned} \gamma _{\epsilon }(x,y; k) = \frac{1}{\epsilon } (k_{\epsilon }(\tilde{x},\tilde{y})-k(x,y)), \end{aligned}$$

where $(\tilde{x},\tilde{y})\in M'(\epsilon )$ is the perturbed version of $(x,y)\in M$. Let $d_M$ be the geodesic distance on $M$. If there exists $C>0$ such that

$$\begin{aligned} \left| \gamma _{\epsilon }(x,y; k) - \gamma _{\epsilon }(x,z; k) \right| \le C_{\alpha } \cdot d_M(y,z)^{\alpha }, \end{aligned}$$

and

$$\begin{aligned} \left| \gamma _{\epsilon }(x,y; k) \right| \le C_{\alpha }, \end{aligned}$$

$\forall x\in M$ and any $\epsilon >0$ in a sufficiently small neighborhood of zero.

Remark

If $k(\cdot ,\cdot ) = d_M(\cdot ,\cdot )$, the infimum of bound $C_{\alpha }$ that makes $k_{\epsilon }$’s perturbed kernel belong to $\alpha$-Hölder class, aka

$$\begin{aligned} C_{\alpha }^* := \sup _{x,y,z,\epsilon } \left\{ \dfrac{\left| \gamma _{\epsilon }(x,y; k) - \gamma _{\epsilon }(x,z; k) \right| }{d_M^{\alpha }(\cdot ,\cdot )}, \left| \gamma _{\epsilon }(x,y; k) \right| \right\} , \end{aligned}$$

is a “natural” quantity indicator of deformations.

In the sequel, we connect $\alpha$-Hölder class of perturbed kernel with perturbed smoothness.

Theorem 6.3

(Main result) Given an integral kernel operator $\mathcal K$ as described in Sect. 2.2, we assume the distance map is in accordance with the geodesics of power $q$, aka, there exist positive constants $C_1$, $C_2$ and some $\delta \ge 0$ such that

$$\begin{aligned} C_1/n^{1+q+\delta } \le \lambda _n(k) \le C_2/n^{1+q},\quad \forall n, \end{aligned}$$

(12)

for some integer $q\ge 1$.

If its perturbed kernels with respect to a class of deformations, i.e., $\gamma _{\epsilon }(\cdot ,\cdot ; k)$, are of $\varepsilon$-Hölder class (Definition 6.2) for some $1\ge \varepsilon > 0$, $\mathcal K$ or the transformed operator $\widetilde{\mathcal K} = \Phi ^T \mathcal K \Phi$ (Sect. 2.2, Eq. 3), as a characterization of the deformed manifold $M'(\epsilon )$, is perturbed $(\alpha ,p)$-smooth (Definition 6.1) as long as

$$\begin{aligned} \left( \frac{1}{2} + \varepsilon \right) p > (1+q+\delta )\alpha + 1. \end{aligned}$$

Thus, spectral signatures (with an arbitrary number of dimensions) of both $\mathcal K$ and $\widetilde{\mathcal K}$ have finite error bounds under smooth manifold deformations (Theorem 6.2).

Proof

For a general Hilbert–Schmidt operator $k$ satisfying uniformly $\varepsilon$-Hölder condition, it has already been shown in [34] that the decay rate of eigenvalues of $k(x,y)$ is at least $O(n^{-1/2-\varepsilon })$. And if $k$ is positive definite, the decay rate is improved to at least $O(n^{-{1}-\varepsilon })$. It immediately follows that for $\varepsilon > \frac{1}{2}$, $\varepsilon$-Hölder continuous kernel $\gamma _{\epsilon }$ is perturbed $(0,1)$-smooth.

For $\alpha >0$, we require the eigenvalues of the characterization kernel that have lower bounds on its decay rate. It is generally not true, for example if $k(x,y)=d_M(x,y)^2$, the resulting Gram matrix has a finite number of nonzero eigenvalues if sample points are uniformly drawn from the Euclidean space. But it is typically tangible, as preliminary work has shown examples where condition (12) holds when $k(x,y)=d_M(x,y)$ (e.g., see [7, 11] examples). Then following the main result on Hölder condition and the decay of eigenvalues [34], one has $\dfrac{\left| \lambda _n(\gamma _{\epsilon }) \right| ^p}{\left| \lambda _n(k) \right| ^{\alpha }}\sim O(n^{-(\frac{1}{2} +\varepsilon )p+\alpha (1+q+\delta )})$.

Appendix B: Spectral convergence of R-BiHDM

Since we know eigenvalues of the Laplace–Beltrami operator follow $\lambda _m \sim \frac{4\pi }{A} m$ for surfaces (Weyl-asymptotic growth of eigenvalue [45]), we can therefore deduce the convergence of a spectral sequence under weak assumptions. Note $\left\| \cdot \right\|$ always denotes the $L_2$-norm of a vector, a function or a matrix, by default.

Lemma 1

Let $A\in \mathbb {C}^{n\times n}$ be an Hermitian matrix, and let $(\hat{\lambda },\hat{x})$ be the computed approximation of an eigenvalue/eigenvector pair $(\lambda ,x)$ of $A$. By defining the residual

$$\begin{aligned} \hat{r} = A \hat{x} - \hat{\lambda }\hat{x}, \hat{x} \ne 0, \end{aligned}$$

it follows that

$$\begin{aligned} \min _{\lambda _i\in \sigma (A)} \left| \hat{\lambda }- \lambda _i \right| \le \dfrac{\left\| \hat{r} \right\| _2}{\left\| \hat{x} \right\| _2}. \end{aligned}$$

Proof

See [32] page 195. $\square$

Theorem 7.1

(Spectral convergence) Let $\widetilde{K}_{m}$ be an $(m+1)\times (m+1)$ square matrix formed by the following steps: (i) take the first $m+1$ rows and first $m+1$ columns of $\widetilde{K}$; (ii) when calculating each $a_j$ in this truncated matrix, every infinite summation in (5) is approximated by the first $m$ terms. Set $\mu _i^{(m)}$ be the $i$-th eigenvalue of $\widetilde{K}_m$, for $i\le m+1$ and $\mu _i{(m)}=0$, for any $i> m+1$. Let $f_m = \sqrt{A}\sum _{i=1}^m\phi _i^2/\lambda _i^2$ converge (pointwisely) to $f$. If $f$ is bounded and square integrable, i.e., eigenpairs $\{(\lambda _i, \phi _i)\}_{i=1}^\infty$ satisfy

$$\begin{aligned} \sum _{i=1}^\infty \dfrac{\left\| \phi _i^2 \right\| ^2}{\lambda _i^4} < \infty , \end{aligned}$$

(13)

it then follows that for any $i$,

$$\begin{aligned} \mu _i^{(m)}\rightarrow \mu _i,\quad \text{ as } m\rightarrow \infty . \end{aligned}$$

Proof

By definition, we have $\widetilde{K}_{m+n} - \widetilde{K}_{m-1}=$

$$\begin{aligned} \sum _{k=m}^{m+n} \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} \frac{2}{\lambda ^2_k}&{}\frac{C_{k,1}}{\lambda _k^2}&{}\ldots &{}\frac{C_{k,m-1}}{\lambda _k^2}&{} 0&{}\ldots &{}a^{(m+n)}_k&{}\ldots \\ \frac{C_{k,1}}{\lambda _k^2}&{}0\\ \vdots &{}&{}\ddots &{}&{}\\ \frac{C_{k,m-1}}{\lambda _k^2}&{}&{}&{}0\\ 0&{}&{}&{}&{}0\\ \vdots &{}&{}&{}&{}&{}\ddots \\ a^{(m+n)}_k&{}&{}&{}&{}&{}&{}-\frac{2}{\lambda ^2_k}\\ \vdots &{}&{}&{}&{}&{}&{}&{}\ddots \end{array}\right] , \end{aligned}$$

where extra rows and columns of zeros are padded to $\widetilde{K}_{m-1}$, and

$$\begin{aligned} C_{m,j} = \sqrt{A}\int _M \phi _m^2\phi _j, \end{aligned}$$

which is the Fourier coefficient of $\sqrt{A}\phi _m^2$ w.r.t series $\{\phi _j\}_{j=1}^{m-1}$. By Bessel’s inequality we have $1+\sum _{j=1}^{m-1}C_{m,j}^2\le A\left\| \phi _m^2 \right\| ^2$, and we also have

$$\begin{aligned} \sum _{k=m}^{m+n}\left| a^{(m+n)}_k \right| ^2\le & {} \sum _{k=m}^{\infty }\left| \int _Mf_{m+n}\phi _k \right| ^2\\\le & {} \left\| f_{m+n} -f \right\| ^2 + \sum _{k=m}^{\infty }\left| \int _M f\phi _k \right| ^2. \end{aligned}$$

By assumptions about $f$ and the norm convergence of Fourier series (as shown for example in [1]), we know that $\left\| f_{m+n} -f \right\| ^2\rightarrow 0$ and $\sum _{k=m}^{\infty }\left| \int _M f\phi _k \right| ^2\rightarrow 0$ as $m\rightarrow \infty$. Since the $L_2$-norm of a matrix is less than its Frobenius norm, we have

$$\begin{aligned}&\!\!\! \left\| \widetilde{K}_{m+n} - \widetilde{K}_{m-1} \right\| ^2_2\\&\quad \le \sum _{k=m}^{m+n}\left( \frac{2}{\lambda _k^4} (4+\sum _{j=1}^{m-1}C^2_{k,j}) + 2\left| a^{(m+n)}_k \right| ^2\right) \\&\quad \le \sum _{k=m}^{m+n}\left( \frac{2}{\lambda _k^4}(3+A\left\| \phi _k^2 \right\| ^2) + 2\left| a^{(m+n)}_k \right| ^2\right) \rightarrow 0 \end{aligned}$$

as $m\rightarrow \infty$. By Lemma 1, we have

$$\begin{aligned} \left| \mu _i^{(m+n)}-\mu _i^{(m-1)} \right|\le & {} \dfrac{\left\| (\widetilde{K}_{m+n} - \widetilde{K}_{m-1})u_i^{(m+n)} \right\| }{\left\| u^{(m+n)}_i \right\| }\\\le & {} \left\| \widetilde{K}_{m+n} - \widetilde{K}_{m-1} \right\| _2 \end{aligned}$$

for any $i<m$, where $u^{(m)}_i$ is the eigenvector associated with $\mu ^{(m)}_i$ of $\widetilde{K}_{m}$. $\{\mu _i^{(m)}\}_{m=1}^{\infty }$ is a Cauchy sequence, that converges. $\square$

Note that the condition, i.e., Eq. (13), used in the spectral convergence theorem is rather weak. Observe that for two-dimensional Riemannian manifold $M$, $f(x)$ is given by the green function $G(x,y)$ with $f(x):=G(x,x)$. If $M$ is a compact manifold (without boundary), it satisfies that $G(x,y)$ is bounded [6], hence $f(x)$ is also bounded for any $x\in M$. So is $\left\| f \right\| ^2$ (square integrable). An experimental validation is also provided in Fig. 7.

Fig. 7

A experimental validation: convergence of the first 30 eigenvalues of an R-BiHDM

Appendix C: Matrices for linear and cubic FEM

Fig. 8

Node configuration of linear and cubic FEM

Table 6 The four matrices of integrals on the unit triangle for linear FEM

Table 7 The four matrices of integrals on the unit triangle for cubic FEM

Here we provide elementary matrices used in linear and cubic FEM. Every FEM has its own set of nodes. There is a finite element associated with every node. The finite element at a specific node is a piecewise polynomial basis function whose value is equal to 1 at the node and 0 at all other nodes. The support of a basis function includes all triangle faces surrounding the node corresponding to the basis function. Consider a triangle face in a mesh with two node configurations shown in Fig. 8. These node configurations are defined for linear and cubic FEM, respectively. Each node in either configuration is associated with a piecewise linear or cubic polynomial basis function. If we focus on a triangle face, an arbitrary bivariate polynomial over the triangle face can be defined as a linear combination of the basis functions associated with the nodes either inside or on the edges of the triangle. This polynomial can interpolate arbitrary prescribed values at the nodes. It reduces to a univariate polynomial along each triangle edge. The two polynomials defined over two adjacent triangles agree with each other along their shared triangle edge. The stiff and mass matrices used in FEM are given as follows.

$$\begin{aligned} L_{ij}(T)= & {} \frac{1}{a(T)}\left( \sum _{k=1}^3 l_k^2(T)J_k\left( t_i,t_j\right) \right) , \quad i,j\in N(T)\\ D_{ij}(T)= & {} a(T)J_4\left( t_i,t_j\right) , \quad i,j\in N(T)\\ L_{ij}(T)= & {} D_{ij}(T) = 0, \quad i,j\not \in N(T) \end{aligned}$$

and

$$\begin{aligned} L = \sum _{T} L(T), \quad D = \sum _{T} D(T), \end{aligned}$$

where $T$ denotes a triangle face, $N(T)$ denotes the set of incidence nodes on $T$, $t_i$ denotes the local index (used for $T$) of the $i$-th node used in the FEM, $l_k(T)$ $(k=1,2,3)$ denotes the length of the $k$-th edge of $T$, $a(T)$ denotes the area of $T$, and $J_k(t_i,t_j)$ denotes the entry with indices $(t_i,t_j)$ of the matrix $J_k$. $J_k$ $(k=1,2,3,4)$ for linear and cubic FEM are given in Tables 6 and 7, respectively. Details on the construction of $J_k$ can be found in [50]. Note that these matrices are assembled triangle-wise rather than element-wise in a typical implementation.