Abstract
This paper investigates a variational model for splines in the image metamorphosis model for the smooth interpolation of key frames in the space of images. The Riemannian manifold of images based on the metamorphosis model defines shortest geodesic paths interpolating two images as minimizers of the path energy which measures the viscous dissipation caused by the motion field and dissipation caused by the material derivative of the image intensity along motion paths. In this paper, we aim at smooth interpolation of multiple key frame images picking up the general observation of cubic splines in Euclidean space which minimize the squared acceleration along the interpolation path. To this end, we propose the spline functional which combines quadratic functionals of the Eulerian motion acceleration and of the second material derivative of the image intensity as the proper notion of image intensity acceleration. We propose a variational time discretization of this model and study the convergence to a suitably relaxed time continuous model via \(\varGamma \)-convergence methodology. As a byproduct, this also allows to establish the existence of metamorphosis splines for given key frame images as minimizers of the time continuous spline functional. The time discretization is complemented by effective spatial discretization based on finite differences and a stable B-spline interpolation of deformed quantities. A variety of numerical examples demonstrates the robustness and versatility of the proposed method in applications. For the minimization of the fully discrete energy functional, a variant of the iPALM algorithm is used.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
At first, we briefly review image metamorphosis as a flexible model for image morphing which generalizes the flow of diffeomorphism approach (cf. the textbook by Younes [46]). It has been extensively investigated by Trouvé, Younes and coworkers [43, 44]. Image metamorphosis is a Riemannian manifold approach to the space of images and can be used to interpolate two images by a connecting geodesic path. Thereby, geodesic paths are minimizers of the associated path energy over all regular paths connecting a pair of input images. The underlying metric associates a cost both to the transport of image intensities via a viscous flow and to image intensity variations along motion paths. The path energy is given as time integral over the metric evaluated on a pair of transport velocities and material derivatives of image intensities. The metamorphosis model has been extended to discrete measures [38], to reproducing kernel Hilbert spaces [39], to functional shapes [11], to images on Hadamard manifolds [17] and to deep feature spaces [16]. Based on the time discrete metamorphosis model proposed in [7], Effland et al. introduced the Bézier curves interpolation in the space of images [19], image geodesics for optical coherence tomography [6] and discrete extrapolation [18]. The (time discrete) metamorphosis model was further used for regularization of several inverse problems in imaging: exemplar-based face colorization [36], sparse and limited angle computerized tomography and super-resolution of images [33], and joint (spatio-temporal) tomographic reconstruction and registration [22].
Geodesic paths are the generalization of straight lines in Euclidean space to Riemannian manifolds. The need for an as smooth as possible interpolation of multiple data points is an extensively studied problem in mathematics. Cubic splines are a widespread and versatile solution for this problem. Due to the famous result by de Boor [9] in Euclidean space, this approach amounts to finding a path \(t\mapsto u(t)\) which minimizes the integral of the squared acceleration \(\int _0^1 |\ddot{u}(t)|^2 \,\textrm{d}t\).
In a Riemannian context, Noakes et al. [35] introduced Riemannian cubic splines as stationary paths of the integrated squared covariant derivative of the velocity. In this paper, we aim at generalizing this ansatz to the Riemannian space of images equipped with the metamorphosis metric. Piecewise geodesic interpolation of images lacks smoothness at the interpolation data points leading to a jerking at the corresponding times when animating the resulting family of images. To overcome this shortcoming, we consider as the spline energy the sum of two integrals over squared acceleration quantities. The first quantity is the classical transport-based Eulerian acceleration, and the second quantity represents the second-order material derivative of the image intensity. Minimizers of this spline energy are one way to naturally generalize Euclidean splines to the metamorphosis model on the space of images. In fact, our model separates in a physically intuitive way the Eulerian flow acceleration and the second material derivative of the image intensity. This way differs from the spline energy introduced by Noakes with the spline energy being the integral over the squared covariant derivative of the path velocity (cf. also [35, 41, 45] or [23]). In this case, the squared covariant derivative of the path velocity with respect to the Riemannian metric would lead to an interwoven model of the different types of acceleration.
One of the first applications of splines in computer vision was work by Mumford [31] where splines appear as a maximum likelihood reconstruction of occluded edges due to the penalization of curvature of arc length parametrized curves. Today, there is a variety of spline approaches in nonlinear spaces and with applications to shape spaces. Trouvé and Vialard [42] studied a second-order shape functional in landmark space based on an optimal control approach. Singh et al. [40] introduced an optimal control method involving a functional which measures the motion acceleration in a flow of diffeomorphisms ansatz for image regression. Tahraoui and Vialard [41] consider a second-order variational model on the group of diffeomorphisms of the interval [0, 1]. They proposed a relaxed model leading to a Fisher–Rao functional, as a convex functional on the space of measures. Vialard [45] showed the existence of a minimizer of the Riemannian acceleration energy on the group of diffeomorphisms endowed with a right-invariant Sobolev metric of high order. Benamou et al. [5] and Chen et al. [12] discussed spline interpolation in the space of probability measures endowed with the Wasserstein metric. Thereby, energy splines are defined as minimizers of the action functional on Wasserstein space which involves the acceleration of measure-valued paths, sharing similarities with the spline functional in the space of images introduced in this paper. The initial computational intractability of such approaches is tackled by a relaxation based on multi-marginal optimal transport and entropic regularization. The transport problem this approach aims at solving might not have a Monge solution, which was remedied by a new method introduced by Chewi et al. [13] to construct measure-valued splines, dubbed transport splines. This method additionally enjoys substantial computational advantages.
In this paper, we will recall the image metamorphosis model and the corresponding path energy whose minimizers are geodesic paths. On that basis, we then derive the novel spline energy functional. Given a set of key frames at disjoint times, a spline curve is then given as a smooth in time minimizer of this spline energy respecting the key frame images as interpolation constraints. Furthermore, we will study a suitable time discrete variational model, which generalizes the time discrete metamorphosis model proposed in [7, 16] . The central contribution of this paper is the convergence of this time discrete model to the time continuous metamorphosis spline model in the sense of Mosco [30]. As a consequence, one obtains existence of metamorphosis spline paths. Finally, we discretize the model in space and we derive a numerical scheme to solve for fully discrete metamorphosis spline paths in the space of images.
Notation. Throughout this paper, we assume that the image domain \(\varOmega \subset \mathbb {R}^d\) for \(d\in \{2,3\}\) is bounded and strongly Lipschitz. We use standard notation for Lebesgue and Sobolev spaces from the image domain \(\varOmega \) to a Banach space X, i.e., \(L^p(\varOmega ,X)\) and \(H^m(\varOmega ,X)\) and omit X if the space is clear from the context. The associated X norms are denoted by \(\Vert \cdot \Vert _{L^p(\varOmega )}\) and \(\Vert \cdot \Vert _{H^m(\varOmega )}\), respectively, and the seminorm in \(H^m(\varOmega )\) is given by \(|\cdot |_{H^m(\varOmega )}\), i.e.,
for \(f\in H^m(\varOmega )\). We use the notation \(C^{k,\alpha }(\overline{\varOmega },X)\) for Hölder spaces of order \(k\ge 0\) with Hölder regularity \(\alpha \in (0,1]\) for the k-th derivatives and the corresponding (semi)norm is
The space \(AC^p([0,1],X)\) is the space of absolutely continuous functions with the derivative in \(L^p((0,1))\).
The symmetric part of a matrix \(A\in \mathbb {R}^{d,d}\) is denoted by \(A^\text {sym}\), i.e., \(A^\text {sym}=\frac{1}{2}(A\!+\!A^\top )\) and the symmetrized Jacobian of a differentiable function \(\phi \) by \(\varepsilon [\phi ]\!=\!(D\phi )^{\text {sym}}\). We denote by \(\mathbbm {1}\) both the identity map and the identity matrix. Finally, \(f_t\) refers to the evaluation of the function at time t, while \(\dot{f}\) refers to the temporal derivative of a differentiable function f.
Organization. This paper is organized as follows. In Sect. 2, we review the most important properties of the flow of diffeomorphism model and the metamorphosis model as its extension. In Sect. 3, the time continuous spline energy is derived and the proper interplay between Lagrangian and Eulerian perspective is discussed. Then, in Sect. 4 a variational time discretization of the continuous spline energy is introduced and the existence of discrete splines is studied. Section 5 represents the time extension of the time discrete quantities, which allows study of convergence to the time continuous model, presented in Sect. 6. Section 7 explains the fully discrete scheme, and Sect. 8 shows how to set up a suitable iPALM algorithm to numerically solve for a spline interpolation given a set of key frames. Finally, Sect. 9 experimentally demonstrates properties of the spline approach for image metamorphosis and shows applications of the proposed method.
In comparison with the conference proceeding [27], we present a significantly more detailed study of both time discrete and time continuous metamorphosis splines. We prove existence of discrete metamorphosis splines, give a suitable continuous extension of discrete metamorphosis splines and show that the discrete spline functional converges in the sense of Mosco to the continuous spline functional. This in particular enables to establish the existence of continuous metamorphosis splines as minimizers of the continuous spline energy. Furthermore, we extended the applications section.
2 Review of Metamorphosis Model
In this section, we briefly review the classical flow of diffeomorphism model and the metamorphosis model as its generalization.
2.1 Flow of Diffeomorphism
The flow of diffeomorphism model [4, 15, 26, 28] is based on Arnold’s paradigm [2] to study of flow of ideal fluids by geodesics on the group of volume-preserving diffeomorphisms. To this end, given a domain \(\varOmega \subset \mathbb {R}^d\), one considers a family of diffeomorphisms \(\left( \psi _t\right) _{t\in [0,1]}:\overline{\varOmega }\rightarrow \mathbb {R}^d\) determined by its time-dependent Eulerian velocity \(v_t=\dot{\psi }_t\circ \psi ^{-1}_t\). The Riemannian structure on this space is given by the metric
and the path energy
Here, L defines a quadratic form corresponding to a higher order elliptic operator. The specific choice, used throughout this paper, is
In this case, the metric \(g_{\psi _t}(\dot{\psi }_t,\dot{\psi }_t)\) describes the viscous dissipation in a multipolar fluid model as investigated by Nečas and Šilhavý [32]. The first term of the integrand represents the dissipation density in a simple Newtonian fluid, and the second term can be regarded as a higher order measure of the fluid friction. Using that the metric \(g_{\psi _t}\) is \(H^m(\varOmega )\)-coercive [15, Theorem 3.1] shows the existence of a flow of diffeomorphisms as a minimizer of the above energy. This minimizer represents a geodesic path connecting two fixed diffeomorphisms.
In the context of image morphing, the flow of diffeomorphism is transporting image intensities along particle paths describing the temporal change of c-channel image intensities \((u_t)_{t \in [0,1]}:\varOmega \rightarrow \mathbb {R}^c\). This transport is given in terms of the equation \(u_t(\cdot ):=u_0\circ \psi ^{-1}_t(\cdot )\) also known as brightness constancy assumption [24], which is equivalent to a vanishing material derivative \(\frac{D}{\partial t}u:=\dot{u}+v\cdot Du\). Given two image intensity functions \(u_A,u_B\), an associated geodesic path is a family of images subject to the constraint \(u_0=u_A\), \(u_1=u_B\) and \(u_t(\cdot )=u_A\circ \psi ^{-1}_t(\cdot )\) where the underlying family of diffeomorphisms \(\left( \psi _t\right) _{t\in [0,1]}\) minimizes the path energy.
2.2 Metamorphosis Model
The metamorphosis approach originally proposed by Miller, Trouvé, Younes and coworkers in [29, 43, 44] generalizes the flow of diffeomorphism model by allowing for image intensity variations along motion paths and penalizing the squared material derivative in the metric. Under the assumption that the image path \(u\) is sufficiently smooth, the metric and the path energy read as
Hence, the flow of diffeomorphism model is the limit case of the metamorphosis model for \(\delta \rightarrow 0\).
Since paths in the space of images do not exhibit any time nor space smoothness properties in general, the evaluation of the material derivative is not well-defined. As a solution to these problems, Trouvé and Younes [43] proposed a nonlinear geometric structure in the space of images \(\mathcal {I}:=L^2(\varOmega ,\mathbb {R}^c)\). In detail, for a given image path \(u\in L^2([0,1],\mathcal {I})\) and an associated velocity field \(v\in L^2((0,1),\mathcal {V})\), where \(\mathcal {V}:=H^{m}(\varOmega ,\mathbb {R}^d)\cap H^{1}_0(\varOmega ,\mathbb {R}^d)\), we define the vector valued weak material derivative \(\hat{z}\in L^2((0,1),L^2(\varOmega ,\mathbb {R}^c))\) by
for a smooth test function \(\eta \in C^{\infty }_c((0,1)\times \varOmega )\). With further consideration of equivalence classes of motion fields and material derivatives inducing the same temporal change of the image intensity, we can consider \((v,\hat{z})\) as a tangent vector in the tangent space of \(\mathcal {I}\) at the image \(u\) and write \((v,\hat{z}) \in T_u\mathcal {I}\). This gives rise to the notion \(H^1([0,1],\mathcal {I})\) for regular paths in the space of images. The path energy in the metamorphosis model for a regular path \(u\in H^1([0,1],\mathcal {I})\) is then defined as
Image morphing of two input images \(u_A,u_B\in \mathcal {I}\) then amounts to computing a shortest geodesic path \(u\in H^1([0,1],\mathcal {I})\) in the metamorphosis model, which is defined as a minimizer of the path energy in the class of regular curves such that \(u(0)=u_A\) and \(u(1)=u_B\). The infimum in (3) is attained, which is shown in [43, Proposition 1 & Theorem 2], and the existence of a shortest geodesic is proven in [43, Theorem 6].
The Lagrangian formulation of variation of the image intensity along motion trajectories is
This motivates a relaxed approach proposed in [16, 17], where the material derivative quantity is retrieved from a variational inequality
for a.e. \(x\in \varOmega \) and all \(1\ge t > s \ge 0\). Formally, the scalar valued \(z=|\hat{z}|\) replaces the actually vector-valued material derivative. In fact, given a path \(u\) in \(L^2([0,1],\mathcal {I})\) the inequality (5) defines a set \(\mathcal {C}(u) \subset L^2((0,1),\mathcal {V})\times L^2((0,1) \times \varOmega )\) of admissible pairs \((v,z) \in \mathcal {C}(u)\) of the velocity and scalar weak material derivative fulfilling this inequality. Then, the geodesic energy for a path \(u\in L^2([0,1],\mathcal {I})\) is defined by
The equivalence of the approaches (3) and (6) follows from [17, Proposition 8], in the sense that for every z satisfying (5) there exists a \(\hat{z}\) satisfying (4) with \(|\hat{z}|\le z\) and for every \(\hat{z}\) satisfying (4) we have that \(z=|\hat{z}|\) satisfies (5). Furthermore, for every \(u \in L^2([0,1],\mathcal {I})\) with non-empty \(\mathcal {C}[u]\) one can show that \(u \in C^0([0,1],\mathcal {I})\) (cf. [16, Remark 1]), which allows point evaluation in time. In both [16, 17], the existence of continuous time geodesic paths was shown, where the relaxed approach turned out to be very natural.
3 Time Continuous Splines in Metamorphosis Model
In this section, we study the spline interpolation for a set of given key frame images in the time continuous setting of metamorphosis model. In mathematical notation, given a set of J key frames \(u_j^I \in \mathcal {I}\) we ask for spline interpolation \((u_t)_{t\in [0,1]}\) which satisfies the constraints
To this end, we recall that cubic splines in Euclidean space minimize the integral over the squared motion acceleration subject to position constraints [9], whereas linear interpolation is associated with the minimization of the integral over the squared motion velocity. In our case, image morphing via minimization of the path energy (2) corresponds to this linear interpolation. Here, we introduce the acceleration quantities which will be penalized in the spline energy. The Eulerian flow acceleration is defined by
and for image paths with enough smoothness the second-order material derivative is given by
As already mentioned in the introduction, this splitting of acceleration term into flow acceleration and second-order change of image intensity does not fully correspond to a Riemannian manifold approach since the stronger interference of the two entities is observed in the covariant derivative of the vector field \((v,\frac{D}{{\partial }t}u)\). The spline energy comprises of the integral over the two quadratic acceleration quantities:
where for the simplicity we use the same elliptic operator (1).
As in the case of geodesic path energy, we give rigorous formulations for general paths \(u \in L^2((0,1),\mathcal {I})\). We observe the pairs \((v,a) \in L^2((0,1), \mathcal {V}) \times L^2((0,1),\mathcal {V})\) which determine the system
and the corresponding diffeomorphic flow \(\psi \in H^2((0,1), H^m(\varOmega ,\varOmega ))\). Notice that from (8), one expects that the acceleration has one derivative less in comparison with the velocity. However, the approach we later take allows us to have the same number of derivatives. A similar gain of smoothness was noticed in [45].
Similar to (4) the Lagrangian formulation of the material acceleration \(\hat{w} \in L^2((0,1),L^2(\varOmega ,\mathbb {R}^c))\) is given by
By further using (4), we have
for all \(s,t \in (0,1)\) and every \(\tau \), such that \(t+\tau , s+\tau \in [0,1]\). Observe that for \(s=t-\tau \) the right hand side of (10) is an integral version of the second-order central difference. Because there is no differentiation involved in these definitions, they work for general image paths. Analogous to (5), we introduce the scalar quantity \(w \in L^2((0,1)\times \varOmega )\) as an relaxation of the weak second-order material derivative
for every \(s \le t \in (0,1), \tau \ge 0, t+\tau \le 1\). This relaxed Lagrangian approach is substantially more handsome in comparison with the Eulerian approach (9) which will be exploited in the proof of consistency of continuous and time discrete approaches.
We can show the equivalence of the two approaches corresponding to (10) and (11) (cf. [17, Proposition 8]).
Proposition 1
For every vector valued \((\hat{z},\hat{w})\) satisfying (4) and (10), there exist scalar quantities (z, w) satisfying (5) and (11) with \(z=|\hat{z}|\) and \(w=|\hat{w}|\). Conversely, for every (z, w) satisfying (5) and (11) there exists \((\hat{z},\hat{w})\) satisfying (4) and (10) with \( z\ge |\hat{z}|\) and \(w\ge |\hat{w}|\).
Proof
The first claim is obvious by the triangle inequality. To prove the converse, let z satisfy (5). We have
from where we conclude that the function \(t \mapsto u_t \circ \psi _t\) is in \(AC^2([0,1],L^2(\varOmega ,{\mathbb {R}^c}))\) which implies the a.e. differentiability and the existence of derivative \({z}' \in L^2((0,1),L^2(\varOmega ,\mathbb {R}^c))\) such that
where \(\hat{z}_r:={z}'_r \circ \psi ^{-1}_r\) and \(|\hat{z}|\le z\) (cf.[1, Remark 1.1.3 & Theorem 1.1.2]). Thus, we have verified (4). This implies
for every \(s \le t \in [0,1]\) and \(\tau >0\) with \(t+\tau \le 1\), and the same holds for integration on interval \([-\tau ,0]\) for \(s-\tau \ge 0\). Taking the limit as \(\tau \) tends to zero and using Lebesgue’s differentiation theorem [21, Theorem 3.21] we conclude that for all \(s \le t \in [0,1]\) and a.e. \(x \in \varOmega \) we have
Thus, we can repeat the above procedure to conclude the existence of \(\hat{w} \in L^2((0,1),L^2(\varOmega ,\mathbb {R}^c))\) satisfying
from where the claim directly follows.
In [23], a regularization of the spline path energy by addition of weighted geodesic path energy was necessary for existence and further analysis of the splines. We follow the analogous approach which seems to be unavoidable also in our model.
Definition 1
(Regularized spline energy) Let \(\sigma >0\), \(m>1+\frac{d}{2}\) be an integer, and \(u \in L^2((0,1),\mathcal {I})\) be an image curve. Then, the regularized spline energy is defined by
where \(\mathcal {C}[u] \subset L^2((0,1), \mathcal {V}) \times L^2((0,1), \mathcal {V}) \times L^2((0,1)\times \varOmega ) \times L^2((0,1)\times \varOmega )\) consists of tuples (v, a, z, w) satisfying
Let us observe that similar to [16, Remark 1] if \(C[u]\) is non-empty, we have an additional regularity of the image curve with \(u \in C^1([0,1],\mathcal {I})\). Motivated by [9], we now define the continuous time spline interpolation for given key frames.
Definition 2
(Continuous time regularized spline interpolation) Let \(\{u_j^I\}_{j=1,\dots ,J} \in \mathcal {I}^J\). We call a minimizer \(u \in L^2((0,1),\mathcal {I})\) of \(\mathcal {F}^\sigma \) that satisfies (7) a continuous time regularized spline interpolation of \(\{u_j^I\}_{j=1,\dots ,J}\) with regularity parameter \(\sigma \).
If we do not impose any additional constraints, we say the continuous time spline interpolation has natural boundary constraints. Imposing periodic boundary condition is equivalent to defining the image curve \(u_t\) on the sphere \(\mathbb {S}^1\) instead of the interval [0, 1]. In the case of Hermite boundary conditions, we additionally fix the values of velocity terms, for both the flow and the image intensity, at the initial and the final time point. Thus, we ask for \(v_0=v_A, v_1=v_B\) and, in light of Proposition 1 and a differentiation of the left hand side of (4), that \(\hat{z}_0=\hat{z}_A\) and \(\hat{z}_1=\hat{z}_B\). Note that in the case of the Hermite boundary conditions (also called clamped boundary conditions), we implicitly require \(t_1=0\) and \(t_{J}=1\), so that \(u_0\) and \(u_1\) are also prescribed.
4 Variational Time Discretization
In this section, we study the variational time discretization of the time continuous (regularized) spline energy. To this end, we pick up the approach of [7, 16] for the variational time discretization of geodesic energy. We consider a discrete image curve \(\textbf{u}=(u_0,\dots ,u_K)\) with \(u_k \in \mathcal {I}\) and define a set of admissible deformations
for a fixed (small) \(\epsilon >0\), which consists of \(C^1(\varOmega ,\varOmega )\)-diffeomorphisms [14, Theorem 5.5-2].
Remark 1
The case \(\epsilon =0\) is discussed in Remark 3.
Considering \(\textbf{u}\in \mathcal {I}^{K+1}\) as time sampling at times \(\frac{k}{K}\), \(k=0,\dots ,K\), of a smooth curve and \(\varvec{\Phi }=(\phi _1,\ldots ,\phi _K)\in \mathcal {D}^K\) as relative flow (\(\phi _k=\psi _{\frac{k}{K}} \circ \psi ^{-1}_{\frac{k-1}{K}}\)) and using forward finite difference approximations we obtain the discrete version of the Eulerian velocity \(v_k:=K(\phi _k -\mathbbm {1})\) and \(\hat{z}_k:=K(u_{k} \circ \phi _{k}-u_{k-1})\) for the discrete material derivative. Furthermore, by using central finite difference we define the discrete acceleration
and
as the discrete version of the second material derivative. Following [7, 16], we consider the discrete path energy
where \(W_D(B)\!:=\!\left| B^{sym}-\mathbbm {1}\right| ^2\) is a simple elastic energy density. Then, the discrete counterpart of the spline energy is defined as
with the energy density \(W_A(B):=\left| B^{sym}\right| ^2\). We note that \(a_k \in H^m(\varOmega ,\mathbb {R}^d)\) by [25, Proposition 2.19]. Finally, the regularized discrete spline energy is given by
As in the continuous time model, we have interpolation constraints. Let \({I^K :=} (i_1,\dots ,i_{J^K})\) be an index tuple with \(2 \le {J^K} \le K\), \(i_j \in \{0,\dots ,K\}\) for \(j=1,\dots ,J^K\). We consider a \({J^K}\)-tuple \(\mathcal {I}^K_{f}=({u}^I_{1},\dots ,{u}^{{I}}_{{J^K}})\) and define the set of admissible image vectors
We are now in a position to define discrete splines.
Definition 3
(Discrete time regularized spline interpolation) Let \(\textbf{u}=(u_0, \dots , u_K) \in \mathcal {I}^{K}_{adm}\). Then, we set
A discrete time regularized spline interpolation of
\(\{{u}^I_{j}\}_{j=1,\dots ,J^K}\) is a discrete \((K+1)\)-tuple that minimizes \(\textbf{F}^{\sigma ,K}\) over all discrete paths in \({\mathcal {I}}^K_{adm}\).
The presented discretization is valid in the case of natural boundary conditions, to which we restrict in further discussions. We remark that in the case of periodic boundary conditions we make an identification \(K \overset{\scriptscriptstyle \wedge }{=}\ 0, K+1 \overset{\scriptscriptstyle \wedge }{=}\ 1\) and the sum in (20) goes up to K. For the discrete version of Hermite boundary conditions, we prescribe \(\phi _1=\overline{\phi }_1, \phi _K=\overline{\phi }_K\), \(u_0=\overline{u}_0, u_K=\overline{u}_K\) and \(\hat{z}_1=\overline{\hat{z}}_1,\hat{z}_K=\overline{\hat{z}}_K\) for given \(\overline{\phi }_1,\overline{\phi }_K\in \mathcal {D}\), \(\overline{u}_0, \overline{u}_K\in \mathcal {I}\) and \(\overline{\hat{z}}_1,\overline{\hat{z}}_K\in L^2(\varOmega ,\mathbb {R}^c)\).
Next, we follow ideas from [16] in order to prove the existence of discrete spline interpolations. The following lemma is the analogous result to [16, Lemma 1], and we only state it for completeness.
Lemma 1
There exists a constant C which only depends on \(\varOmega , m,d, \gamma \) such that
for all \(\phi \in \mathcal {D}\) satisfying \(\int _\varOmega W_D(D\phi ) \!+\! \gamma |D^{m} \phi |^2 \,\textrm{d}x \le C_{\phi }\).
Proof
An application of the Gagliardo–Nirenberg inequality [34] yields
The last term in (23) is bounded by
To estimate the lower order term on the right hand side, we use Korn’s and Poincare’s inequality and write
Thus, the lemma follows by combining (23), (24) and (25).
Remark 2
The analogous result holds for the boundedness of acceleration, i.e., \(\int _\varOmega W_A(Da)+\gamma |D^{m}a|^2 \,\textrm{d}x \le C_a\) implies \(\Vert a\Vert _{H^{m}(\varOmega )}\le C\sqrt{C_a}\).
Now, we show the well-posedness of (22).
Proposition 2
For every \(K \in \mathbb {N}\) and every image vector \(\textbf{u}=(u_0,\dots ,u_K) \in {\mathcal {I}^K_{adm}}\), there exists a deformation vector \(\varvec{\Phi }=(\phi _1,\dots ,\phi _K) \in \mathcal {D}^K\) such that
Proof
Let \(\{\varvec{\Phi }^j\}_{j \in \mathbb {N}} \in \mathcal {D}^K\) be a sequence for which it holds \(\lim _{j \rightarrow \infty }\!\textbf{F}^{\sigma ,K,D}[\textbf{u},\varvec{\Phi }^j]\!=\! \inf _{{\tilde{\varvec{\Phi }}} \in \mathcal {D}^K} \!\textbf{F}^{\sigma ,K,D}[\textbf{u},{\tilde{\varvec{\Phi }}}]\) and \(\textbf{F}^{\sigma ,K,D}[\textbf{u},\varvec{\Phi }^{j}] \le \overline{\textbf{F}^{\sigma ,K}} :=\textbf{F}^{\sigma ,K,D}[\textbf{u},\mathbbm {1}^{K}]\). By Lemma 1, we have
Thus, \(\{\varvec{\Phi }^j\}_{j \in \mathbb {N}}\) is uniformly bounded in \(H^m(\varOmega ,\varOmega )^K\). By reflexivity of this space, there exists a subsequence (with the same label) such that \(\varvec{\Phi }^j \rightharpoonup \varvec{\Phi }\) in \(H^m(\varOmega ,\varOmega )^K\). By the compact Sobolev embedding, we have \(\varvec{\Phi }^j \rightarrow \varvec{\Phi }\) in \(C^{1,\alpha }(\overline{\varOmega },\overline{\varOmega })^K\) for \(\alpha \in (0,m-1-\frac{d}{2})\), which gives us that \(\varvec{\Phi }\in \mathcal {D}^K\). Analogously, we have boundedness of \({\{\textbf{a}^j\}_{j \in \mathbb {N}}}\) in \(H^{m}(\varOmega ,\varOmega )^{K-1}\) and thus, a convergent subsequence satisfying \(\textbf{a}^j \rightharpoonup \textbf{a}\) in \(H^m(\varOmega ,\varOmega )^{K-1}\) and \(\textbf{a}^j \rightarrow \textbf{a}\) in \(C^{1,\alpha }(\overline{\varOmega },\overline{\varOmega })^{K-1}\). Here, for every \(j \in \mathbb {N}\), we have \(\textbf{a}^j=(a^j_1,\dots ,a^j_{K-1})\) given by (18). From the strong convergence of deformations, we have that \(a_k=K^2(\phi _{k+1} \circ \phi _k - 2 \phi _k + \mathbbm {1})\) for all \(k=1,\dots ,K-1\). Using weak lower semicontinuity of \(H^m\)-seminorm and continuity of energy densities, we have for all k, as \(j \rightarrow \infty \):
To handle the rest of the terms, we show that for all \(u \in \mathcal {I}\) we have \(u \circ \phi ^j \rightarrow u \circ \phi \) in \(\mathcal {I}\) for \(\{\phi ^j\}_{j \in \mathbb {N}} \in \mathcal {D}\) with \(\phi ^j \rightarrow \phi \) in \(C^1(\overline{\varOmega },\overline{\varOmega })\) as \(j \rightarrow \infty \). To see this, we approximate \(u\) by smooth functions \(\{\tilde{u}^i\}_{{i \in \mathbb {N}}}\in C^\infty (\overline{\varOmega },\overline{\varOmega })\) with \(\Vert u-\tilde{u}^i\Vert _{\mathcal {I}}\rightarrow 0\). Then, using the transformation formula we obtain
By first choosing i and then j, we have that this expression converges to 0. Hence, for every \(k=1,\dots ,K\) we have \(\Vert u_k \circ \phi ^j_k - u_{k-1}\Vert _\mathcal {I}\rightarrow \Vert u_k \circ \phi _k - u_{k-1}\Vert _\mathcal {I}\). Furthermore, via nested application of (27) \(\Vert u_{k+1} \circ \phi ^j_{k+1} \circ \phi ^j_k - 2 u_k \circ \phi ^j_k + u_{k-1}\Vert _{\mathcal {I}} \rightarrow \Vert u_{k+1} \circ \phi _{k+1} \circ \phi _k - 2 u_k \circ \phi _k + u_{k-1}\Vert _{\mathcal {I}}\) for \(k=1,\dots ,K-1\), which together with (26) finishes the proof.
In the next step, under suitable conditions, we prove that there exists a minimizing vector in \(\mathcal {I}^K_{adm}\) (see (21)) for a fixed deformation vector \(\varvec{\Phi }\in \mathcal {D}^K\).
Proposition 3
Let \(K \ge 2\), \(\mathcal {I}^K_{f}\) and \(\varvec{\Phi }\in \mathcal {D}^K\) be fixed. Assume that the deformations satisfy, for every \(x \in \varOmega \),
Then, there exists a vector of images \(\textbf{u}\in \mathcal {I}^K_{adm}\) such that
Proof
Let \({\{\textbf{u}^j\}}_{j \in \mathbb {N}} \in \mathcal {I}^K_{adm}\) be an approximation sequence such that
Here, \(\overline{\textbf{F}^{\sigma ,K,D}}:=\textbf{F}^{\sigma ,K,D}[\mathbf {\overline{u}},\varvec{\Phi }]\) represents a finite upper bound for the energy with the vector of images \(\mathbf {\overline{u}}\) satisfying \(\overline{u}_k=u^I_1\) for \({0 \le k \le i_1},\) \(\overline{u}_k=u^I_{j}\) for \(i_j \le k \le i_{j+1}\) with \(1 \le j \le J^K-1\) and \(\overline{u}_k=u^I_{J^K}\) for \( i_{J^K} \le k \le K\). Indeed, we have
where we used the transformation formula and (28). By further use of (28), we have
from where we have, by induction, that \({\{u^j_k\}_{j \in \mathbb {N}}}\) is uniformly bounded in \(\mathcal {I}\) for every \(k=0,\dots ,K\). By reflexivity, there exists a subsequence (labeled in the same way) such that \(u^j_k \rightharpoonup u_k\) for some \(\textbf{u}\in \mathcal {I}^K_{adm}\). It remains to verify the weak lower semicontinuity of the matching functional, i.e., we have to show that
for every \(k=1,\dots ,K\) and \(k=1,\dots ,K-1\), respectively. To this end, we first show \(u_k^j\circ \phi _k\rightharpoonup u_k\circ \phi _k\) in \(\mathcal {I}\). For every \(v \in \mathcal {I}\), the transformation formula yields
which converges to 0 since \((v(\det (D\phi _k))^{-1})\circ \phi _k^{-1}\in \mathcal {I}\) due to (28). Hence, \(u_k^j\circ \phi _k-u_{k-1}^j\rightharpoonup u_k\circ \phi _k-u_{k-1}\) in \(\mathcal {I}\), which readily implies (30) and by applying the same technique in a nested fashion we get (31). Altogether, we have
from where the optimality follows.
We are now in the position to show the existence of discrete time spline interpolations.
Theorem 1
(Existence of discrete time spline interpolations) Let \(K\ge 2\). Then, for every \(\mathcal {I}^K_{f}\) there exists \(\textbf{u}\in \mathcal {I}^K_{adm}\) such that
Proof
Consider a sequence xxxx \(\{\textbf{u}^j\}_{j \in \mathbb {N}}\) for which it holds \(\lim _{j \rightarrow \infty } \textbf{F}^{\sigma ,K}[\textbf{u}^j]=\inf _{\textbf{v} \in \mathcal {I}^K_{adm}} \textbf{F}^{\sigma ,K}[\textbf{v}] \le \overline{\textbf{F}^{\sigma ,K}}\), Here, \(\overline{u}_k=\overline{u}({\frac{k}{K}},\cdot )\) for \(k=0,\dots ,K\), where \(\overline{u}\) is a smooth in time curve with \(\bar{u}({\frac{i_j}{K}},\cdot )=u^I_j\) for every \(j=1,\dots ,J^K\). Then, we have
where the constant is independent of K. Furthermore, for every j let \(\textbf{F}^{\sigma ,K}[\textbf{u}^j]=\textbf{F}^{\sigma ,K,D}[\textbf{u}^j,\varvec{\Phi }^j]\), by Proposition 2. As in the proof of Proposition 2, we have weak convergence of \({\{\varvec{\Phi }^j\}_{j \in \mathbb {N}}}\) in \(H^{m}(\varOmega ,\varOmega )^K\) and strong in \(C^{1,\alpha }(\overline{\varOmega },\overline{\varOmega })^K\) to some \(\varvec{\Phi }\in \mathcal {D}^K\). Furthermore, we again have \(a^j_k \rightharpoonup a_k\) in \(H^{m}(\varOmega ,\varOmega )\), and strongly in \(C^{1,\alpha }(\overline{\varOmega },\overline{\varOmega })\), where \(a_k= K^2(\phi _{k+1} \circ \phi _k -2\phi _k + \mathbbm {1})\) and estimates from (26) are satisfied. By Proposition 3, we may replace \(\textbf{u}^j\) by an energy optimal image vector. Keeping the same label and following the same arguments as in (29), we conclude that \({\{\textbf{u}^j\}_{j \in \mathbb {N}}}\) is uniformly bounded in \(\mathcal {I}\), weakly converging to \(\textbf{u}\). We are left to verify the estimates
for every \(k=1,\dots ,K\) and \(k=1,\dots ,K-1\), respectively. To that end, it is enough to show \(u^j_k\circ \phi ^j_k \rightharpoonup u_k\circ \phi _k\) in \(\mathcal {I}\). To see this, we first take into account the decomposition
The second term is handled as in (27). It remains to consider the convergence properties of the first term. For a test function \(v\in \mathcal {I}\), we obtain using the transformation rule
The right hand side converges to 0 due to the convergence \((\det (D\phi ^j_k))^{-1}\circ (\phi ^j_k)^{-1}\rightarrow \det (D\phi _k))^{-1}\circ \phi _k^{-1}\) in \(L^\infty (\varOmega )\), \(v \circ \phi _k^j \rightarrow v \circ \phi _k\) in \(\mathcal {I}\) and \(u_k^j \rightharpoonup u_k\) in \(\mathcal {I}\) for \(j\rightarrow \infty \). This proves our claim and finally proves the theorem.
Remark 3
The results of this section remain valid for any \(W_D\) satisfying conditions \((W1)-{(W2)}\) from [16]. Furthermore, let us observe the case when \(i_j=K \cdot t_j\) and \(u_{i_j}=u^I_j\) for \(t_j \in [0,1]\) and \(u^I_j \in \mathcal {I}\) for every \(j=1,\dots ,J\) (cf. (7) and (21)). Then, for every large enough K (depending on \(u^I_j\) and \(t_j\)) one can show the existence of the discrete time spline interpolation even if \(\epsilon =0\) in the definition of the admissible set (17). Namely, by (32) we have that \(\overline{\textbf{F}^{\sigma ,K}}\) is a fixed finite upper bound for the discrete spline energy, independent of K. Then, using Lemma 1 and Sobolev embedding theorem we have
By Lipschitz continuity of the determinant, we have
for large enough K, which proves \(\min \limits _{k=1,\dots ,K} \det (D\phi _k) \ge c_{\det }>0\) and we can proceed as before.
5 Temporal Extension Operators
In this section, we define the suitable time extensions of the time discrete quantities from the previous section. This is necessary to compare discrete and continuous spline functional and to study convergence.
Let \(K \ge 2\), \(\tau =\frac{1}{K}\), \(t^K_k=k\tau \), \(t^K_{k\pm \frac{1}{2}}=(k \pm \frac{1}{2})\tau \), \(k=0,1,\dots ,K\). Furthermore, consider a vector of images \(\textbf{u}^K=(u_0^K,\dots ,u^K_K) \in \mathcal {I}^{K+1}\) and a vector of deformations \(\varvec{\Phi }^K=(\phi ^K_1,\dots ,\phi ^K_K) \in \mathcal {D}^K \). We first define a discrete incremental transport path \(y^K\) with \(y^K_t=y^K_0(t,\cdot )\) for \(t \in [0,t^K_{\frac{1}{2}}{]}\), \(y^K_t=y^K_k(t,\cdot )\) for \(t \in {(}t^K_{k-\frac{1}{2}},t^K_{k+\frac{1}{2}}{]}\) with \(k=1,\dots ,K-1\) and \(y^K_t=y^K_K(t,\cdot )\) for \(t \in {(}t^K_{K-\frac{1}{2}},1]\), where
and, for \(k=1,\ldots ,K-1\)
This can be seen as the cubic Hermite interpolation on intervals \({(}t^K_{k-\frac{1}{2}},t^K_{k+\frac{1}{2}}]\), and an affine interpolation on \([0,t^K_{\frac{1}{2}}{]}\) and \((t^K_{K-\frac{1}{2}},1]\). In particular, observe that \(y^K_{t^K_{k-\frac{1}{2}}}=\tfrac{\mathbbm {1}+\phi ^K_k}{2}\) and \(y^K_{t^K_{k+\frac{1}{2}}}=\tfrac{(\mathbbm {1}+\phi ^K_{k+1})\circ \phi ^K_k}{2}\) with the corresponding slopes \(\tfrac{\phi ^K_k-\mathbbm {1}}{\tau }\) and \(\tfrac{(\phi ^K_{k+1}-\mathbbm {1})\circ \phi ^K_k}{\tau }\), respectively (cf. Fig. 1 (left) for a sketch).
We define the image extension operator \(\mathcal {U}^K[\textbf{u}^K,\varvec{\Phi }^K] \in L^2([0,1],\mathcal {I})\) as \(\mathcal {U}^K[\textbf{u}^K,\mathbf {\Phi }^K](t,x)=u^K(t,x)\) where
and, for \(k=1,\dots ,K-1\) and \(t \in {(}t^K_{k-\frac{1}{2}},t^K_{k+\frac{1}{2}}{]}\)
This can be seen as blending between the “half-way images” \(\tfrac{u^K_{k-1}+u^K_k \circ \phi ^K_k}{2}\) and \(\tfrac{(u^K_{k}+u^K_{k+1} \circ \phi ^K_{k+1})\circ \phi ^K_k}{2}\) along the incremental transport path \(y^K\). This is depicted in Fig. 1 right.
Left: Schematic drawing of the Hermite interpolation (blue) on the time interval \([(k-\tfrac{1}{2})/K, (k+\tfrac{1}{2})/K]\) together with the discrete acceleration \(a_k^K(x)\). Right: Image extension \(\mathcal {U}^K[\textbf{u}^K,\varvec{\Phi }^K](\cdot , x)\) along a path \((\psi _t^K(x))_{t \in [0,1]}\) from the left, plotted against time. Dots represent the values \(u_k^K\), and crosses the “half-way” values \(\tfrac{1}{2} (u_k^K+u_{k-1}^K)\) along the discrete transport path (Color figure online)
The discrete velocity field corresponding to the discrete incremental transport path \(y^K_t\) is given by
for \(t \in [0,t^K_{\frac{1}{2}}{]}\) and \(t \in {(}t^K_{K-\frac{1}{2}},1]\), respectively, and
for \(t \in {(}t^K_{k-\frac{1}{2}},t^K_{k+\frac{1}{2}}{]}\) with \(k=1,\dots ,K-1\). The corresponding discrete acceleration is \(a_0^K(t,\cdot )=a_K^K(t,\cdot ){:=}0\) for \(t \in [0,t^K_{\frac{1}{2}}{]}\) and \(t \in {(}t^K_{K-\frac{1}{2}},1]\), respectively, and
for \(t \in {(}t^K_{k-\frac{1}{2}},t^K_{k+\frac{1}{2}}{]}\) with \(k=1,\dots ,K-1\). The spatial inverse of the discrete incremental transport path is denoted by \(x^K_k(t,\cdot ), ~k=0,\dots ,K\) which, following [14, Chapter 5] exists if \(\max _{k=1,\dots ,K} \Vert D\phi ^K_k- \mathbbm {1}\Vert _{C^0(\varOmega )} <c\) for a small enough constant \(c>0\) (see Sect. 6, (39), (40)). We define the velocity and the acceleration along the incremental transport path by
Now, the actual discrete flow given as the map \((t,x) \mapsto \psi _t^K(x)\) is defined recursively by
One can directly show that (37) is well-defined in the sense of equations (13)- (14), i.e.,
Based on this, the first-order scalar weak material derivative of \(u^K\) can be defined as the absolute value of the material derivative along the paths \(t \mapsto \psi _t^K(x)\) with
and
for \(t \in {(}t^K_{k-\frac{1}{2}},t^K_{k+\frac{1}{2}}{]},~k=1,\dots ,K-1\). For the second-order scalar weak material derivative, we have that for \(t \in {(}t^K_{k-\frac{1}{2}},t^K_{k+\frac{1}{2}}{]},~k=1,\dots ,K-1\)
and \(w^K_t{:=}0\) elsewhere, which is the absolute value of the second time derivative of \(\mathcal {U}^{K}[\textbf{u}^K,\varvec{\Phi }^K]\) along the path \(t \mapsto \psi _t^K(x)\). Indeed, one easily verifies (cf. [17, Proposition 9]) that \(z^K_t\) and \(w^K_t\) are admissible in the sense of equations (15) and (16), i.e.,
Remark 4
For periodic boundary conditions, we make the cubic interpolation definition on \(t \in {(}t^K_{k-\frac{1}{2}},t^K_{k+\frac{1}{2}}{]}\) for \(k=1,\dots ,K\), with the convention \(K \overset{\scriptscriptstyle \wedge }{=}\ 0, K+1 \overset{\scriptscriptstyle \wedge }{=}\ 1\) and \({(}t^K_{K-\frac{1}{2}},t^K_{K+\frac{1}{2}}{]}= {(}t^K_{K-\frac{1}{2}},1] \cup [0,t^K_{\frac{1}{2}}{]}\).
Finally, we define the extension of the energy \(\textbf{F}^{\sigma ,K}\) to a functional \(\mathcal {F}^{\sigma ,K}\) by
if there exists such \(\textbf{u}^K,\varvec{\Phi }^K\) and \(+\infty \) else. The existence of the infimum follows from the continuity of the constraint \(\mathcal {U}^{K}[\textbf{u}^K,\varvec{\Phi }^K]=u\) w.r.t. the weak convergence of \(u^K\) and the strong convergence of \(\varvec{\Phi }^K\)(cf. [17, Lemma 11]).
6 Convergence of Time Discrete Splines
In this section, we study the convergence of extension of discrete regularized spline energy \(\mathcal {F}^{\sigma ,K}\) to continuous counterpart \(\mathcal {F}^\sigma \) and the convergence of time discrete minimizers to a time continuous minimizer in the sense of Mosco.
We prove the Mosco convergence of energy \(\mathcal {F}^{\sigma ,K}\) to \(\mathcal {F}^\sigma \) in \(L^2((0,1){, \mathcal {I}})\).
Theorem 2
(Mosco-convergence of the discrete spline energies) Let \(\sigma >0\). Then, the discrete spline energies \(\{\mathcal {F}^{\sigma ,K}\}_{K \in \mathbb {N}}\) converge to \(\mathcal {F}^{\sigma }\) in the sense of Mosco in the topology \(L^2((0,1){, \mathcal {I}})\) for \(K \rightarrow \infty \). In explicit
-
(i)
for every sequence \(\{u^K \in L^2((0,1){, \mathcal {I}})\}_{K \in \mathbb {N}}\) which converges weakly to \(u\in L^2((0,1){, \mathcal {I}})\) as \(K \rightarrow \infty \), it holds \(\liminf _{K \rightarrow \infty } \mathcal {F}^{\sigma ,K}[u^K] \ge \mathcal {F}^{\sigma }[u]\),
-
(ii)
for every \(u\in L^2((0,1){, \mathcal {I}})\), there exists a sequence \(\{u^K\}_{K \in \mathbb {N}}\) such that \(u^K \rightarrow u\) in \(L^2((0,1){, \mathcal {I}})\) as \(K\!\rightarrow \!\infty \) and \(\limsup _{K \rightarrow \infty } \mathcal {F}^{\sigma ,K}[u^K] \le \mathcal {F}^{\sigma }[u]\).
Remark 5
The above result holds for any choice of \(W_D\) satisfying assumption \((W1)\!-\!(W3)\) from [16], though we restrict ourselves to the special case \(W_D\!=\!|A^{sym}-\mathbbm {1}|^2\).
Remark 6
In the case of periodic boundary conditions, the same applies in the topology \(L^2(\mathbb {S}^1{, \mathcal {I}})\). The proof requires minor alterations implied by already indicated changes in energy and interpolation.
6.1 Proof of the \(\liminf \) Estimate
Suppose we have a sequence \(\{u^K \in L^2((0,1),\mathcal {I})\}_{K \in \mathbb {N}}\) such that \(u^K \rightharpoonup u\) in that space as \(K \rightarrow \infty \), and we suppose that \(\mathcal {F}^{\sigma ,K}[u^K]< \overline{\mathcal {F}} < \infty \). By definition for every K large enough an image vector \(\textbf{u}^K \in \mathcal {I}^{K+1}\) and a corresponding optimal vector of deformations \(\varvec{\Phi }^K \in \mathcal {D}^K\) exist, such that
For the image vectors \(\textbf{u}^K\) and the deformation vectors \(\varvec{\Phi }^K\), we define the discrete velocity and acceleration quantities as in Sect. 4 and their extensions as in Sect. 5. Using Lemma 1, we have
which implies that \(y^K_k\) is invertible for any \(k=0,\dots ,K\) and any K large enough. Thus, we are able to define all the temporal extended quantities introduced in the previous section. Furthermore, for every \(t \in [0,1]\) we have that \(y^K_k(t,\cdot )\) is in \(C^{1,\alpha }(\overline{\varOmega },\overline{\varOmega })\) and it is bounded uniformly in t, and by [8, Theorem 2.1] the same holds for \(x^K_k(t,\cdot )\) with an estimate
In what follows, the quantities \(\hat{z}^K_k\) and \(\hat{w}^K_k\) are time discrete quantities from Sect. 4, while \(z^K_t\) and \(w^K_t\) are their time extensions from Sect. 5. The estimate (41) together with locally Lipschitz property of the determinant function gives
Using the same ideas, together with \(|z^K_t - |\hat{z}^K_k \circ x^K_{k,t}||\le \frac{1}{K} |\hat{w}^K_k \circ x^K_{k,t}|\) (cf. (38)) on every subinterval we have
This implies the uniform boundedness of \(\{z^K\}_{K \in \mathbb {N}}\) and \(\{w^K\}_{K \in \mathbb {N}}\) in \(L^2((0,1) \times \varOmega )\) and by reflexivity the existence of weakly convergent subsequences (with the same labelling) to z and w, respectively. Then, by weak lower semicontinuity we have
Using Korn’s and Poincare’s inequality, we get
The analogous estimates hold for \(v^K\) and \(D^m v^K\), where we additionally use \(\left| v^K_t-|v^K_k|\right| \le \frac{1}{K} |{a}^K_k|\) on every subinterval (cf. (35)). Hence, we have that \(\{v^K\}_{K \in \mathbb {N}}\) and \(\{a^K\}_{K \in \mathbb {N}}\) are uniformly bounded in \(L^2((0,1),\mathcal {V})\) and they have corresponding weak limits v and a in that space.
Furthermore, we compute the Taylor expansion of \(W_A((t-t^K_{k-\frac{1}{2}})^2Da^K_k)\) at \(t^K_{k-\frac{1}{2}}\), evaluated at \(t=t^K_{k+\frac{1}{2}}\) to get
For the remainder term, we have \(r_{a,k}^K=\mathcal {O}(K^{-6}|Da^K_k|^3)\) and by using Lemma 1 and (40)
Then, we use weak lower semicontinuity of the energy to write
Analogous Taylor expansion arguments give
Altogether, (42), (45) and (46) give
We are left to show that the limit objects v, a, z, w are indeed corresponding quantities for the image curve \(u\). First, let us observe that by using (41) (cf. also [20, Proposition 1.2.4,1.2.7]) we have
where \(\tilde{v}^K_t\) and \(\tilde{a}^K_t\) are introduced in (36). This implies that we have the uniform boundedness of \(\{\tilde{v}^K\}_{{K \in \mathbb {N}}}\) in \(L^2((0,1),\mathcal {V})\) and in \(L^2((0,1),C^{1,\alpha }(\overline{\varOmega },\mathbb {R}^d))\) by (47). Then by [17, Theorem 6] and Hölder’s inequality we can infer that \(\{\psi ^K\}_{K \in \mathbb {N}}\) is uniformly bounded in \(C^{0,\frac{1}{2}}([0,1],C^{1,\alpha }(\overline{\varOmega },\overline{\varOmega }))\). Furthermore, by compact embedding of Hölder spaces, we have for some \(\min \{\frac{1}{2},\alpha \}>\beta >0\) that \(\psi ^K \rightarrow \psi \) in \(C^{0,\beta }([0,1],C^{1,\beta }(\overline{\varOmega },\overline{\varOmega }))\). To show that \(\psi \) is indeed the solution corresponding to v, we consider \({\psi }^{v^K}\), the solution corresponding to \(v^K\). By weak continuity of the solution operator mapping velocities to the flows [17, Theorem 6], we have \({{\psi }^{v^K}} \rightarrow \tilde{\psi }\) in \(C^0([0,1]\times \overline{\varOmega })\). Furthermore, by [17, Remark 7] we have the Lipschitz continuity of the solution operator and using the spatial Lipschitz property of \(v^K_k\) together with (39) we have \(\psi ^K-{{\psi }^{v^K}} \rightarrow 0\) in \(C^0([0,1] \times \overline{\varOmega })\) which finally confirms \(\psi =\tilde{\psi }\).
To show that the equation \(\ddot{\psi }_t=a_t \circ \psi _t\) is satisfied first observe that the equation \(\ddot{\psi }^K_t=\tilde{a}^K_t\circ \psi ^K_t\) ensures the uniform boundedness of \(\{\ddot{\psi }^K\}_{{K\in \mathbb {N}}}\) in the space \(L^2((0,1),C^{1,\alpha }(\overline{\varOmega },\overline{\varOmega }))\). To this end, we used the uniform boundedness of \(\{\tilde{a}^K\}_{{K\in \mathbb {N}}}\) in the space
\(L^2((0,1),C^{1,\alpha }(\overline{\varOmega },\overline{\varOmega }))\)(cf. (48)) and \(\{\psi ^K\}_{{K\in \mathbb {N}}}\) in
\(C^0([0,1],C^{1,\alpha }(\overline{\varOmega },\overline{\varOmega }))\). This together with the previous paragraph implies that \(\{\psi ^K\}_{{K\in \mathbb {N}}}\) is uniformly bounded in \(H^2((0,1),C^{1,\alpha }(\overline{\varOmega },\overline{\varOmega }))\) and converges weakly to \(\psi \) in \(H^2((0,1),C^{1,\beta }(\overline{\varOmega },\overline{\varOmega }))\) for \(0<\beta <\frac{1}{2} \min \{\frac{1}{2},\alpha \}\). Altogether, we have that \(\psi \in H^2((0,1),C^{1,\beta }(\overline{\varOmega },\overline{\varOmega }))\) and
for all \( \theta \in C^{\infty }_c((0,1) \times \varOmega )\). Hence, it sufficies to verify \(\tilde{a}^K \circ \psi ^K \rightharpoonup a \circ \psi \) in \(L^2((0,1)\times \varOmega )\). Since we already have \(a^K \circ \psi ^K \rightharpoonup a \circ \psi \) in \(L^2((0,1)\times \varOmega )\), we conclude the proof by checking that \(\tilde{a}^K \circ \psi ^K - a^K \circ \psi ^K \rightarrow 0\) in \(L^2((0,1),C^0(\overline{\varOmega },\overline{\varOmega }))\). Indeed,
where we used the Lipschitz property of \(a^K_t\), the transformation formula and finally (39).
In order to show that z and w are indeed scalar weak material derivatives of u, first observe that from the weak convergence \(u^K \rightharpoonup u\) and the strong convergence \(\psi ^K \rightarrow \psi \) by similar approximation arguments as for (33) we obtain \(u^K \circ \psi ^K \rightharpoonup u \circ \psi \) in \(L^2((0,1) \times \varOmega )\) and analogously \(z^K \circ \psi ^K \rightharpoonup z \circ \psi \) and \(w^K \circ \psi ^K \rightharpoonup w \circ \psi \). Next, note that for \(s,t \in [0,1]\) we obtain
where we used Hölder’s inequality, the transformation formula and uniform boundedness of \((\psi ^K)^{-1}\). Thus, \(\{u^K \circ \psi ^K\}_{K \in \mathbb {N}}\) is a subset of
for some finite \(L>0\). Then, by the weak closedness of \(A_{\frac{1}{2},L}\) we obtain \(u\circ \psi \in A_{\frac{1}{2},L}\). Then, for every \(\tilde{\varOmega } \subset \varOmega \) the functional \(u\mapsto \int _{\tilde{\varOmega }} |u_t(x)-u_s(x)| \,\textrm{d}x\) is continuous because the point evaluation in time is continuous and since it is convex on \( A_{\frac{1}{2},L}\), this implies weak lower semicontinuity and we obtain
Since this holds for every \(\tilde{\varOmega } \subset \varOmega \), one obtains that z is the first-order scalar material derivative for \(u\). We prove that w is the second scalar weak material derivative for \(u\) in an analogous way. This finally finishes the proof of the \(\liminf \)-inequality.
As a preparatory step for the proof of the \(\limsup \)-estimate, we state a corollary of the preceding proof.
Proposition 4
For \(u \in L^2([0,1],\mathcal {I})\) with \(\mathcal {F}^\sigma [u] < \infty \), there exists an optimal tuple \((v,a,z,w) \in \mathcal {C}[u]\) such that
Proof
The functional \(\mathcal {F}^\sigma \) is coercive by Korn’s inequality and Gagliardo–Nirenberg interpolation estimate, and it is clearly weak lower semicontinuous. Since \(\mathcal {C}[u]\) is a subset of a reflexive Banach space, then we just have to show weak closedness of the set. This is verified as above.
6.2 Proof of the \(\limsup \) Estimate and the Construction of a Recovery Sequence
Consider an image curve \(u \in L^2([0,1],\mathcal {I})\) with finite regularized spline energy. Then, the previous proposition guarantees the existence of an associated optimal velocity field, an acceleration field and the first- and second-order weak material derivatives, denoted by \((v,a,z,w)\in \mathcal {C}[u]\), respectively, i.e.,
We define
where \(\psi _t\) is the flow associated with velocity v and \(\psi _0=\mathbbm {1}\). We have
by Lemma 1 and Cauchy’s inequality. For the second inequality, we used [10, Lemma 3.5] which states that
Thereby, for K large enough we have \(\varvec{\Phi }\in \mathcal {D}^K\) and we are in a position to define
where the point time evaluation is possible since \(u \in C^{{1}}([0,1],\mathcal {I})\) (cf. [16, Remark 1]).
In what follows, we present more detailed arguments for the second-order terms, while the arguments for the first-order terms follow as in [17, Theorem 14].
First, we are able to relate the discrete and the continuous second-order material derivative by
Here, we first used (49) and the transformation formula for the first and the second equality, respectively, then the definition of the second-order material derivative (11) for the first inequality, and the Cauchy–Schwarz inequality and (50) in the last two estimates. Summing the above expressions over \(k=1,\ldots , K-1\), we obtain
Next, we express \(a^K_k\) in terms of a:
where in the second equality we used (49), and in the last equality (14). Then, using the Cauchy–Schwarz inequality and (51) we obtain the following estimate
The same Taylor expansion argument as in (43) and (44) now implies, together with (53)
Applying Jensen’s inequality twice on L, and taking into account the above relation between \(a_k^K\) and a gives
We now estimate the summands of L individually. For the first term, we use that \(|{\text {}}{tr}(AB)|\!\le \! |{\text {}}{tr}(A)|\!+\!|{\text {}}{tr}A(B-\mathbbm {1})|\), (50) and transformation rule to get
For the second term, we use (51) and the fact for any \(0\le \tilde{m} \le m\) and \(f \in H^m(\varOmega ,\mathbb {R}^d), g \in H^{\tilde{m}}(\varOmega ,\mathbb {R}^d)\) we have \(\Vert fg\Vert _{H^{\tilde{m}}(\varOmega )}\le C \Vert f\Vert _{H^m(\varOmega )}\Vert g\Vert _{H^{\tilde{m}}(\varOmega )}\) [25, Lemma 2.3] to get
and iterating this argument and using the transformation formula we have
Altogether with (53) and (54), we have
which together with (52) gives
This readily implies the \(\limsup \)-inequality for the pure spline part of the functional \(\mathcal {F}^{\sigma ,K}\).
Following analogous steps as above, we can fully repeat the procedure from [17, Theorem 14] to obtain the estimate for the lower order path energy
which finally proves the \(\limsup \)-inequality.
We are left to show that \(u^K \rightarrow u\) in \(L^2((0,1){,\mathcal {I}})\) as \(K \rightarrow \infty \). To this end, we introduce the piecewise constant interpolation
Then, for \(t \in (t^K_{k-1}, t^K_{k-\frac{1}{2}}]\) with \(k=1,\dots ,K-1\) we estimate
where we used the transformation formula, (50), (52) and an analogous estimate for the first-order material derivative. For every \(t \in (0,1)\), we can find a sequence \(\{k(K)\}_{K \in \mathbb {N}}\) such that for any K large enough \(t \in (t^K_{k(K)-\frac{1}{2}}, t^K_{k(K)+\frac{1}{2}}]\). We uniformly approximate the sequence \(\{u^K_{k(K)}\}_{K \in \mathbb {N}}\) by smooth functions (cf. (27)) and use (50) and (53) to prove
Plugging this back above we have that \(u^K - \bar{u}^K \rightarrow 0 \) in \(L^\infty ([0,1],\mathcal {I})\) as \(K \rightarrow \infty \). On the other hand, as \(\{\bar{u}^K\}_{K \in \mathbb {N}}\) is a sequence of piecewise constant approximations of \(u \in C^1([0,1],\mathcal {I})\) we have that \(\bar{u}^K \rightarrow u\) in \(L^2([0,1],\mathcal {I})\) as \(K \rightarrow \infty \). Altogether, \(u^K \rightarrow u\) in \(L^2((0,1),\mathcal {I})\) as we wanted to show. This concludes the proof of \(\limsup \) estimate and the construction of recovery and thus concludes the proof of Mosco convergence.
As a corollary of the previous theorem, we are able to show the existence of the continuous time spline defined in Sect. 3. To this end, let \(J\ge 2\) and \((t_1,\ldots ,t_J)\subset [0,1]\cap \mathbb {Q}\) be a sequence of fixed times. Then, for infinitely many \(K\in \mathbb {N}\) one can choose \(i_j^K:=K\cdot t_j\in \mathbb {N}\) for all \(j=1,\ldots ,J\). Let \(({u}^I_j)_{j={1},\ldots ,J}\subseteq \mathcal {I}\) be the set of constraint images at the corresponding constraint times.
Theorem 3
(Convergence of discrete spline interpolations to continuous ones) For every K that satisfies the above condition, let \(u^K \in L^2([0,1],\mathcal {I})\) be a minimizer of \(\mathcal {F}^{\sigma ,K}\) among the image curves satisfying \(u^K=\mathcal {U}^{K}[\textbf{u}^K,\varvec{\Phi }^K]\) with \(u^K_{i_j^K}={u}^I_j\) for all \(j \in \{{1},\dots ,J\}\). Then, a subsequence of \(\lbrace u^K\rbrace _{K\in \mathbb {N}}\) converges weakly in \(L^2([0,1],\mathcal {I})\) to a minimizer of the continuous spline energy \(\mathcal {F}^\sigma \) as \(K\rightarrow \infty \). This minimizer satisfies \(u_{{t_j}}={u}^I_j\) for all \(j \in \{{1},\dots ,J\}\), and the associated sequence of discrete energies converges to the minimal continuous spline energy.
Proof
For \(j=1,\dots ,J\) let \(\eta ^j:[0,1]\rightarrow \mathbb {R}\) be smooth functions with \(\eta ^j(t_i)=\delta _{ij}\). We define a smooth interpolating curve of the fixed images \(\tilde{u}(t):=\sum _{j=0}^J\eta ^{j}(t){u}^I_j\). Let us define the vector \(\tilde{\textbf{u}}^K:=(\tilde{u}(t^K_{1}),\ldots , \tilde{u}(t_K^K))\) and its time extension \(\tilde{u}^K:=\mathcal {U}^{K}[\tilde{\textbf{u}}^K,\mathbbm {1}^K]\). This image curve gives an admissible candidate for a minimizer of the functional \(\mathcal {F}^{\sigma ,K}\). Namely,
where the upper bound is independent of K. As defined above, let \(\{u^K:=\mathcal {U}^K[\textbf{u}^K,\varvec{\Phi }^K]\}_{K \in \mathbb {N}}\), where \(\textbf{u}^K,\varvec{\Phi }^K\) are optimal pairs for the discrete spline (see Theorem 1). In particular, we have \(\mathcal {F}^{\sigma ,K}[u^K]=\textbf{F}^{\sigma , K}[\textbf{u}^K,\varvec{\Phi }^K]<\overline{\mathcal {F}}\). As in (29), we show uniform boundedness of \(u^K_k\) in \(L^2(\varOmega )\). This further implies, together with boundedness of the deformations and the convergence of the discrete transport paths to the identity that \(u^K_t\) is uniformly bounded in \(L^2(\varOmega )\). Therefore, \(\lbrace u^K\rbrace _{K\in \mathbb {N}}\) is uniformly bounded in \(L^\infty ([0,1],\mathcal {I})\) and a subsequence converges weakly to some \(u\in L^2([0,1],\mathcal {I})\).
Now, we follow the usual argument and assume that there exists an image path \(\hat{u}\in L^2([0,1],\mathcal {I})\) with finite energy such that \(\mathcal {F}^\sigma [\hat{u}]<\mathcal {F}^\sigma [u].\) By the \(\limsup \)-part of Theorem 2, there exists a sequence \(\lbrace \hat{u}^K\rbrace _{K\in \mathbb {N}}\subseteq L^2((0,1),\mathcal {I})\) such that \(\limsup _{K\rightarrow \infty } \mathcal {F}^{\sigma ,K}[\hat{u}^K]\le \mathcal {F}^\sigma [\hat{u}]\). Now, we apply the \(\liminf \)-part of Theorem 2, thus obtaining
which is a contradiction to the above assumption. Hence, u minimizes the continuous spline energy over all admissible image curves and discrete spline energies converge to the limiting spline energy along a subsequence, i.e., \(\lim _{K\rightarrow \infty }\mathcal {F}^{\sigma ,K}[u^K]=\mathcal {F}^\sigma [u]\), which follows from (56) by using \(\hat{u}=u\).
Finally, we show that \(u_{t_j}=u^I_j\). To this end recall that for the sequence of piecewise constant approximations \(\{\bar{u}^K\}_{K \in \mathbb {N}}\) given by (55), we showed that \(u^K_t-\bar{u}^K_t \rightarrow 0\) in \(\mathcal {I}\), uniformly in t. Together with \(u^K_t \rightharpoonup u_t\) in \(\mathcal {I}\) for every \(t \in [0,1]\) which is showed by a trace theorem type argument (see [7, Theorem 4.1, (iv)]), we have the needed result since \(\bar{u}^K_{t_j}={u}^I_j\).
The analogous result for arbitrary \((t_1,\ldots ,t_J)\subset [0,1]\) follows from density of \(\mathbb {Q}\) in [0, 1]. Let us remark that in light of Proposition 1 we have that for optimal scalar quantities z, w it holds \(z=|\hat{z}|\) and \(w=|\hat{w}|\).
7 Fully Discrete Metamorphosis Splines
To numerically implement splines for image metamorphosis, we have to further discretize the space. Here, we present a model for c image channels and a two dimensional image domain \(\varOmega :=[0,1]^2\) and follow the fully discrete version of image metamorphosis introduced in [16]. Before presenting the details of the space discretization, to avoid double warping in the second material derivative term (19) and to further increase the robustness of the model we explicitly introduce a vector valued material derivative \(\bar{z} \in L^2((0,1),L^2(\varOmega ,\mathbb {R}^{c}))\). This leads to a relaxation of (12):
with a penalty on the misfit of the new variable \(\bar{z}\) and the actual material derivative \(\hat{z}\) given by (4), while w is the material derivative of \(\bar{z}\). The time discrete counterpart \(\textbf{F}^{\sigma ,K}[\textbf{u}]\) is defined by
where, for \(\bar{\textbf{z}} = (\bar{{z}}_1,\ldots , \bar{{z}}_K)\) we write
Here, \(\bar{{w}}_k=K(\bar{z}_{k+1} \circ \phi _k - \bar{z}_{k})\) is the discrete material derivative of \(\bar{z}_{k}\), while \(\hat{z}_k=K(u_k \circ \phi _k - u_{k-1})\) is the actual material derivative of \(u_k\), with the corresponding second-order derivative \((\hat{w}_k)_{k=1}^{K-1}\) given by (19).
For \(M,N \ge 3\) we define the computational domain
with discrete boundary \(\partial \varOmega _{\scriptscriptstyle {MN}}{:=}\varOmega _{\scriptscriptstyle {MN}}\cap \partial ([0,1]^2) \) and \(\left\| \textbf{u}\right\| _{L^p_{\scriptscriptstyle {MN}}}^p{:=}\frac{1}{\scriptscriptstyle {MN}}\sum _{({\textbf{x},\textbf{y}})\in \varOmega _{\scriptscriptstyle {MN}}}\Vert \textbf{u}({\textbf{x},\textbf{y}})\Vert _p^p\). The discrete image space is \(\mathcal {I}_{\scriptscriptstyle {MN}}:=\{\textbf{u}:\varOmega _{\scriptscriptstyle {MN}}\rightarrow \mathbb {R}^{c}\}\), and the set of admissible deformations is
where the discrete Jacobian operator of \(\varvec{\phi }\) at \({(\textbf{x},\textbf{y})}\in \varOmega _{\scriptscriptstyle {MN}}\) is defined as the forward finite difference operator with Neumann boundary conditions. Here and in the rest of the paper, we used bold faced letters for fully discrete quantities. A spatial warping operator \(\textbf{T}\) that approximates the pullback of an image channel \(\textbf{u}^j\circ \varvec{\phi }\) at a point \({(\textbf{x},\textbf{y})}\in \varOmega _{\scriptscriptstyle {MN}}\) is defined by
where s is the third-order B-spline interpolation kernel. This form of warping is also used for composition of deformations, i.e., we define the fully discrete acceleration as an approximation of (18) by
In summary, the fully discrete spline energy in the metamorphosis model for a \((K+1)\)-tuple \((\textbf{u}_k)_{k=0}^K\) of discrete images, a K-tuple \((\bar{\textbf{z}}_k)_{k=1}^{K}\) of discrete derivatives and a K-tuple \((\varvec{\phi }_k)_{k=1}^K\) of discrete deformations reads as
where
While in the spatially continuous context the compactness induced by the \(H^{m}\)-seminorm is indispensable, in this fully discrete model grid dependent regularity is ensured by the use of cubic B-splines. Thus, we dropped the higher order Sobolev norm terms in this fully discrete model.
To improve the robustness of the overall optimization, we take into account a multiresolution strategy. In detail, on the coarse computational domain of size \(M_{L}\times N_{L}\) with \(M_{L}=2^{-(L-1)}M\) and \(N_{L}=2^{-(L-1)}N\) for a given \(L\ge 1\), a time discrete spline sequence \({(\textbf{u}_k)_{k=0}^{K}}\) is computed as minimizer of \(\textbf{F}_{M_{L} N_{L}}^{\sigma ,K}\) subject to given fixed images \(\textbf{u}_{i_j}=\textbf{u}^I_{j}, ~ j=1,\dots , J\). In subsequent prolongation steps, the width and the height of the computational domain are successively doubled and the initial deformations, images and derivatives are obtained via a bilinear interpolation of the preceding coarse scale solutions.
8 Numerical Optimization Using the iPALM Algorithm
In this section, we discuss the numerical solution of the above fully discrete variational problem based on the application of a variant of the inertial proximal alternating linearized minimization algorithm (iPALM, [37]). Using this algorithm effective optimization results was achieved for a wide range of non-convex and non-smooth problems. In particular, it was already used for numerical optimization in the context of the deep feature metamorphosis model [16]. Following [16], to enhance the stability the warping operation is linearized with respect to the deformation at \(\varvec{\phi }^{[\beta ]}\in \mathcal {D}_{\scriptscriptstyle {MN}}\) coming from the previous iteration which leads to the modified energies
with
To further stabilize the computation, the Jacobian operator applied to the images is approximated using a Sobel filter. Here, \(\langle \cdot , \cdot \rangle \) represent the pointwise product of the involved matrices. We use the proximal mapping of a functional \(f:\mathcal {D}_{\scriptscriptstyle {MN}} \rightarrow (-\infty ,\infty ]\) for \(\tau >0\) given as
Then, with the function values on \(\partial \varOmega _{\scriptscriptstyle {MN}}\) remaining unchanged, the proximal operator we are interested in is given by
where \(\varLambda ^s_j:=\varLambda ({\bar{\textbf{z}}}_{k}^{j},{\bar{\textbf{z}}}_{k+1}^{j},\varvec{\phi }_{k}^{[\beta ]})\) and \(\varLambda ^g_j:=\varLambda (K\textbf{u}_{k-1}^j+{\bar{\textbf{z}}}_{k}^{j},K\textbf{u}_{k}^{j}, \varvec{\phi }_k^{[\beta ]})\). The first terms in both brackets are activated only for \(k<K\).
Left: Time discrete spline with framed key frame images (first row), color-coded displacement field (second row), discrete second-order material derivative (third row) and color-coded discrete acceleration field (fourth row) for the Gaussians example and values of the parameters \(\delta =5\cdot 10^{-3}\), \(\sigma =1\), \(\theta =5\cdot 10^{-5}\). The colors and their intensities indicate the direction and the intensity of the field, as indicated by the color wheel on the left. Right: Euclidean splines in (x, y, m) coordinates for the input parameters versus splines for metamorphosis in (x, y, m) extracted from the numerical results in post-processing, with (x, y) denoting the center of mass and m the mass of the distribution
Left: Time discrete spline (top row) and piecewise geodesic (middle row) interpolation with framed key frames. The bottom row shows the difference in intensity between the different interpolations, using the color map \(-\) 0.35 
0.35. Right: Width of the interpolated shape measured at the horizontal axis of symmetry (in number of pixels) for a spline interpolation (orange) and piecewise geodesic interpolation (green) showing the concavities (\(\delta =5\cdot 10^{-3}\), \(\sigma =1\), \(\theta =5\cdot 10^{-4}\)) (Color figure online)
Time discrete spline with framed fixed images (first and second row), first-order material derivative slack variable \(\bar{\textbf{z}}\) (third and fourth row), second-order material derivative with energies comparison (fifth and sixth row) and color-coded acceleration field with energies comparison (seventh and eight row) for values of the parameters \(\delta =2\cdot 10^{-2},\,\sigma =2,\,\theta =8\cdot 10^{-4}\). The graphics on the right in row four and six show for the spline (orange) time plots of the \(L^2\)-norm of the actual second-order material derivative \(\hat{w}_{k} \) and the dissipation energy density reflecting motion acceleration \(\Vert W_A(\nabla _{\scriptscriptstyle {MN}}\textbf{a}_k)\Vert _{L^1_{\scriptscriptstyle {MN}}}\), respectively. This is compared to the corresponding piecewise geodesic interpolation (green) (not visualized here, cf. Fig. 7)
The actual minimization \(\textbf{F}_{\scriptscriptstyle {MN}}^{\sigma ,K}\) is performed by Algorithm 1. We used the following notation for the extrapolation with \(\beta >0\) of the \(k^{th}\) path element in the \(i^{th}\) iteration step
while the acceleration \(\textbf{a}^{[\beta ,{i}]}\) is computed with correspondingly updated \(\varvec{\phi }^{[\beta ,{i}]}\) values. Furthermore, \(I^K\) is the set of fixed indices (cf. (21)) and we denote by L[h] the Lipschitz constant of the gradient of the function h, which is determined by backtracking [3]. The discrete deformations are initialized by the identity deformation, the discrete images by piecewise linear interpolation of the key frame images, while the discrete derivatives are initialized as differences of two consecutive images in the sequence.
9 Applications
In what follows, we investigate and discuss qualitative properties of the spline interpolation in the space of images, being aware that the superior temporal smoothness of this interpolation is difficult to show with series of still images. For all the examples, we use \({L=}5\) levels in the multi-level approach and \(I=250\) iterations of iPALM algorithm on each level with the extrapolation parameter \(\beta =\frac{1}{\sqrt{2}}\). For the first two examples \({M=}N=64\), while for the others \({M=}N=128\). Also, for the first two and the final example \(K=8\), while for the others \(K=16\). For plotting of images, we crop the values to [0, 1], for the material derivative the values are scaled to the interval [0, 1] for plotting, while for the displacement and acceleration plots hue refers to the direction and the intensity is proportional to its norm as indicated by the color wheel.
Figure 2 shows a first test case. As key frames, we consider three images showing two-dimensional Gaussian distribution with small variance at different positions and of different mass. Spline interpolation is compared with piecewise geodesic interpolation. Furthermore, it is depicted that for the metamorphosis spline, the curve in (x, y, m)-space (position, mass) corresponds almost perfectly to the cubic spline interpolation of the parameters of the Gaussian distribution on the key frames.
As a next step, we conceptually compare splines for metamorphosis and piecewise geodesic paths in Fig. 3. For this specific example, we considered the image of a circle and two identical squares as key frames. The influence of the circle’s curvature on the spline segment between the two identical squares is visible via concave ’edges’, while for the piecewise geodesic interpolation any memory of the circle is naturally lost in the constant interpolation between the two squares.
Left: Time discrete spline with framed fixed images (top and bottom row). Right: Energy density norm of acceleration flow \(\Vert W_A(\nabla _{\scriptscriptstyle {MN}}\textbf{a}_k)\Vert _{L^1_{\scriptscriptstyle {MN}}}\) (top), and \(L^2\)-norm of the actual second-order material derivative \(\hat{w}_k\) (bottom). Parameter values: \(\delta =10^{-3}, \sigma =2, \theta =2\cdot 10^{-5}\)
Top and bottom rows: Two time discrete splines with key-frames images differing in shape and color. Parameter values: \(\delta =8 \cdot 10^{-3}, \sigma =2.5, \theta =2\cdot 10^{-4}\). For visualization purposes, only even-numbered frames \(\textbf{u}_k\) are depicted
Next, we consider spline interpolation between three human portraits in Fig. 4. The plots of second material derivative and acceleration show a strong concentration around the key frames, where the spline is expected to be smooth and the piecewise geodesic path just Lipschitz.
The analogous observations hold for Fig. 5.
The impact of the coloring of the key frame images on the geometry of the spline interpolation and the interplay between the Eulerian flow acceleration and the second-order material derivative along motion paths is depicted in Fig. 6. Therein, we consider block-colored letters as key-frame images. In our first example, the coloring is consistent with the interpolating flow shown in the black and white spline interpolation in Fig. 5. Hence, in the corresponding spline interpolation color is mainly passively transported along this flow rather than blending it (top row). In the second example, the red patch in the middle key-frame image is chosen to be located top right instead, further away from the red patches in the extremal key-frame images (bottom row). Hence, the coloring is no longer consistent to the flow in the black and white example. In fact, the transport in particular of the blue color is now strongly reconfigured as seen in the first interval between the ‘P’ and the ‘A’. Furthermore, in the second interval between the ‘A’ and the ‘Q’ blending from blue to red and from red to blue occurs.
A comparison of splines and piecewise geodesic paths is shown in Fig. 7. One particularly observes that for the faces the shown spline image is thicker and for the letters the spline image shows more round contours than the for the piecewise geodesic counterpart. This is again the non-local impact of key frames beyond those bounding the current interpolation.
Top row: Image \(\textbf{u}_{11}\) for the human face example, second-order material derivative \(\hat{\textbf{w}}_k\) and acceleration field \(\textbf{a}_k\) for \(k=8\), for the time discrete piecewise geodesic (left image of each panel pair) and spline (right image of each panel). The pairs of material derivatives and the acceleration fields are jointly scaled to reflect the differences in intensities. Bottom row: Same visualization of image \(\textbf{u}_4\) from the letter example
Next, we ask for a reconstruction of frames given certain frames at selected time stamps extracted from a video. Here, we compare the resulting spline interpolation and the piecewise geodesic interpolation directly with corresponding frames of the original video as a benchmark for both approaches. Indeed, Fig. 8 shows this comparison of the original frames, spline interpolation and piecewise geodesic interpolation for the images extracted from a video made by David Rogers from Vanderbilt University in the 1950s,Footnote 1 which shows the interaction between white blood cells and bacteria. The spline interpolation clearly shows less blending artifacts in comparison with the piecewise geodesic interpolation.
First row: The original frames extracted from the video showing white blood cell (neutrophil) chasing a Staphylococcus aureus bacterium. The images are courtesy of Robert A. Freitas, Institute for Molecular Manufacturing, California, USA. Second row: Time discrete spline with framed fixed images. Third row: Time discrete piecewise geodesic with framed fixed images. Fourth row: Fully discrete second-order material derivative \(\hat{w}_k\) for the discrete spline interpolation. Fifth row: Fully discrete second-order material derivative \(\hat{w}_k\) for the discrete piecewise geodesic interpolation. The values of parameters are \(\delta =4\cdot 10^{-2}\), \(\sigma =2.5\), \(\theta =1.6 \cdot 10^{-4}\)
10 Conclusion
In this paper, we introduced a novel model for the smooth interpolation of a set of given key frame images based on the image metamorphosis model. This is achieved by means of a minimization of the spline energy functional. Thereby, one penalizes acceleration reflected in the underlying transport and Eulerian acceleration defined in terms of the second-order change of image intensity along the flow paths. Based on [7, 16], we proposed a variational time discretization by approximation of the aforementioned quantities by finite differences. We prove consistency of this time discrete model to the time continuous counterpart by the means of Mosco convergence. Space discretization is based on finite differences and approximation of warping operation by cubic B-splines, while for numerical optimization we use the iPALM algorithm. We present a range of numerical experiments where we compare the differences between the piecewise geodesic interpolation and our novel spline interpolation in the space image. The quantitative results show that the spline interpolation achieves significant improvement with respect to the smoothness in time, which is indicated in better control and regularity of acceleration quantities in the vicinity of the key frames. Furthermore, the influence of the preceding and/or succeeding key frame(s) is visible for the spline interpolation, which naturally relates to the larger support of the spline basis functions for Euclidean cubic splines. Still open is an investigation of the differences between our spline model based on the splitting of acceleration quantities and the model based on the Riemannian second-order covariant derivative as relevant acceleration quantity. Furthermore, a challenge is the adaptation of our approach to textures and feature based image representations.
References
Ambrosio, L., Gigli, N., Savaré, G.: Gradient Flows: In Metric Spaces and in the Space of Probability Measures. Springer, Berlin (2008)
Arnold, V.: Sur la géométrie différentielle des groupes de Lie de dimension infinie et ses applications à l’hydrodynamique des fluides parfaits. Ann. Inst. Fourier (Grenoble) 16(1), 319–361 (1966)
Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)
Beg, M.F., Miller, M.I., Trouvé, A., Younes, L.: Computing large deformation metric mappings via geodesic flows of diffeomorphisms. Int. J. Comput. Vis. 61(2), 139–157 (2005)
Benamou, J.D., Gallouët, T.O., Vialard, F.X.: Second-order models for optimal transport and cubic splines on the Wasserstein space. Found. Comput. Math. 19(5), 1113–1143 (2019)
Berkels, B., Buchner, M., Effland, A., Rumpf, M., Schmitz-Valckenberg, S.: GPU based image geodesics for optical coherence tomography. In: Bildverarbeitung für die Medizin, Informatik aktuell, pp. 68–73. Springer (2017). https://doi.org/10.1007/978-3-662-54345-0_21
Berkels, B., Effland, A., Rumpf, M.: Time discrete geodesic paths in the space of images. SIAM J. Imaging Sci. 8(3), 1457–1488 (2015)
Bojarski, B., Hajłasz, P., Strzelecki, P.: Sard’s theorem for mappings in Hölder and Sobolev spaces. Manuscripta Math. 118(3), 383–397 (2005)
de Boor, C.: Best approximation properties of spline functions of odd degree. J. Math. Mech. 12, 747–749 (1963)
Bruveris, M., Vialard, F.X.: On completeness of groups of diffeomorphisms. J. Eur. Math. Soc. 19(5), 1507–1544 (2017)
Charon, N., Charlier, B., Trouvé, A.: Metamorphoses of functional shapes in Sobolev spaces. Found. Comput. Math. 18(6), 1535–1596 (2018)
Chen, Y., Conforti, G., Georgiou, T.T.: Measure-valued spline curves: An optimal transport viewpoint. SIAM J. Numer. Anal. 50(6), 5947–5968 (2018)
Chewi, S., Clancy, J., Le Gouic, T., Rigollet, P., Stepaniants, G., Stromme, A.: Fast and smooth interpolation on Wasserstein space. In: International Conference on Artificial Intelligence and Statistics, pp. 3061–3069. PMLR (2021)
Ciarlet, P.G.: Mathematical elasticity. Vol. I, Studies in Mathematics and its Applications, vol. 20. North-Holland Publishing Co., Amsterdam (1988)
Dupuis, P., Grenander, U., Miller, M.I.: Variational problems on flows of diffeomorphisms for image matching. Quart. Appl. Math. 56(3), 587–600 (1998)
Effland, A., Kobler, E., Pock, T., Rajković, M., Rumpf, M.: Image morphing in deep feature spaces: Theory and applications. J. Math. Imaging Vis. 63(2), 309–327 (2021)
Effland, A., Neumayer, S., Rumpf, M.: Convergence of the time discrete metamorphosis model on Hadamard manifolds. SIAM J. Imaging Sci. 13(2), 557–588 (2020)
Effland, A., Rumpf, M., Schäfer, F.: Image extrapolation for the time discrete metamorphosis model: Existence and applications. SIAM J. Imaging Sci. 11(1), 834–862 (2018). https://doi.org/10.1137/17m1129544
Effland, A., Rumpf, M., Simon, S., Stahn, K., Wirth, B.: Bézier curves in the space of images. In: Proc. of International Conference on Scale Space and Variational Methods in Computer Vision, Lecture Notes in Computer Science, vol. 9087, pp. 372–384. Springer, Cham (2015)
Fiorenza, R.: Hölder and locally Hölder Continuous Functions, and Open Sets of Class \(C^{k}, C^{k, \lambda }\). Birkhäuser (2017)
Folland, G.B.: Real Analysis: Modern Techniques and their Applications, vol. 40. Wiley, London (1999)
Gris, B., Chen, C., Öktem, O.: Image reconstruction through metamorphosis. Inverse Prob. 36(2), 025001 (2020)
Heeren, B., Rumpf, M., Wirth, B.: Variational time discretization of Riemannian splines. IMA J. Numer. Anal. 39(1), 61–104 (2018)
Horn, B.K., Schunck, B.G.: Determining optical flow. Artif. Intell. 17(1–3), 185–203 (1981)
Inci, H., Kappeler, T., Topalov, P.: On the regularity of the composition of diffeomorphisms. Mem. Amer. Math. Soc. 226(1062), vi–60 (2013)
Joshi, S.C., Miller, M.I.: Landmark matching via large deformation diffeomorphisms. IEEE Trans. Image Process. 9(8), 1357–1370 (2000)
Justiniano, J., Rajković, M., Rumpf, M.: Splines for Image Metamorphosis. In: Scale Space and Variational Methods in Computer Vision: 8th International Conference, SSVM 2021, Virtual Event, May 16–20, 2021, Proceedings, pp. 463–475. Springer (2021)
Miller, M.I., Trouvé, A., Younes, L.: On the metrics and Euler-Lagrange equations of computational anatomy. Annu. Rev. Biomed. Eng. 4(1), 375–405 (2002)
Miller, M.I., Younes, L.: Group actions, homeomorphisms, and matching: a general framework. Int. J. Comput. Vis. 41(1–2), 61–84 (2001)
Mosco, U.: Convergence of convex sets and of solutions of variational inequalities. Adv. Math. 3, 510–585 (1969)
Mumford, D.: Elastica and computer vision. In: Algebraic geometry and its applications, pp. 491–506. Springer (1994)
Nečas, J., Šilhavý, M.: Multipolar viscous fluids. Quart. Appl. Math. 49(2), 247–265 (1991)
Neumayer, S., Persch, J., Steidl, G.: Regularization of inverse problems via time discrete geodesics in image spaces. Inverse Prob. 35(5), 055005–28 (2019). https://doi.org/10.1088/1361-6420/ab038a
Nirenberg, L.: An extended interpolation inequality. Ann. Scuola Norm. Sup. Pisa Cl. Sci. (3) 20, 733–737 (1966)
Noakes, L., Heinzinger, G., Paden, B.: Cubic splines on curved spaces. IMA J. Math. Control Inform. 6(4), 465–473 (1989)
Persch, J., Pierre, F., Steidl, G.: Exemplar-based face colorization using image morphing. J. Imaging 3(4), 666 (2017)
Pock, T., Sabach, S.: Inertial proximal alternating linearized minimization (iPALM) for nonconvex and nonsmooth problems. SIAM J. Imaging Sci. 9(4), 1756–1787 (2016)
Richardson, C.L., Younes, L.: Computing metamorphoses between discrete measures. J. Geom. Mech. 5(1), 131–150 (2013)
Richardson, C.L., Younes, L.: Metamorphosis of images in reproducing kernel Hilbert spaces. Adv. Comput. Math. 42(3), 573–603 (2016)
Singh, N., Vialard, F.X., Niethammer, M.: Splines for diffeomorphisms. Med. Image Anal. 25(1), 56–71 (2015)
Tahraoui, R., Vialard, F.X.: Minimizing acceleration on the group of diffeomorphisms and its relaxation. ESAIM Control Optim. Calc. Var. 25, 6669 (2019)
Trouvé, A., Vialard, F.X.: Shape splines and stochastic shape evolutions: a second order point of view. Quart. Appl. Math. 70(2), 219–251 (2012)
Trouvé, A., Younes, L.: Local geometry of deformable templates. SIAM J. Math. Anal. 37(1), 17–59 (2005)
Trouvé, A., Younes, L.: Metamorphoses through Lie group action. Found. Comput. Math. 5(2), 173–198 (2005)
Vialard, F.X.: Variational second-order interpolation on the group of diffeomorphisms with a right-invariant metric. In: S. Kushnarev, A. Qiu, L. Younes (eds.) Mathematics Of Shapes and Applications, pp. 1–14. World Scientific (2020)
Younes, L.: Shapes and Diffeomorphisms, Applied Mathematical Sciences, vol. 171. Springer, Berlin (2010)
Acknowledgements
This work was partially supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) via project 211504053 - Collaborative Research Center 1060 and project 390685813 - Hausdorff Center for Mathematics. The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.
Funding
Open Access funding enabled and organized by Projekt DEAL.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This publication is an extended version of the previous conference proceeding [27] presented at SSVM 2021.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Justiniano, J., Rajković, M. & Rumpf, M. Consistent Approximation of Interpolating Splines in Image Metamorphosis. J Math Imaging Vis 65, 29–52 (2023). https://doi.org/10.1007/s10851-022-01128-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10851-022-01128-5