A Variational Model for Gradient-Based Video Editing

Abstract

In this work we present a gradient-based variational model for video editing, addressing the problem of propagating gradient-domain information along the optical flow of the video. The resulting propagation is temporally consistent and blends seamlessly with its spatial surroundings. In addition, the presented model is able to cope with additive illumination changes and handles occlusions/dis-occlusions. The problem of propagation along the optical flow arises in different video editing applications. In this work we consider the application where a user edits a frame by modifying the texture of an object’s surface and wishes to propagate this editing throughout the video.

Appendices

Appendix: 1 An Occlusion Detection Method

In this section we discuss a two-step simple method to detect occlusions given a dense optical flow. In the first step we aim to identify regions which are potentially occluded. In the second step we categorize the previously selected regions into occluded and still visible. Figure 25 shows an example of the two-step occlusion detection method.

Fig. 25

The proposed two-step occlusion handling. For clarity, we show only the processing related to the forward optical flow from $\varOmega _{t-1}$ to $\varOmega _t.$ Figures a and b show two consecutive frames. c Shows the selection result of the first step described in Sect. . d Shows the selection result after the second step described in Sect. . Figure 7 shows the output without and with occlusion handling respectively

The problem of occlusion detection is intrinsic to the optical flow problem. In fact, some optical flow algorithms estimate occlusions as part of the estimation of the movement (Alvarez et al. 2007; Ayvaci et al. 2012). Such algorithms output an occlusion mask together with the optical flow.

Many optical flow algorithms however, do not provide occlusion masks. To be able to work with such optical flows, we describe a simple method to detect occlusions. Any other occlusion detection method could be used instead.

1.1 1.1 First Step, the Detection of Potentially Occluded Regions

Let us consider two adjacent frames $t-1$ and $t.$ In an ideal case the forward optical flow from $t-1$ to $t$ should map points in $t-1$ that exist in $t,$ to their corresponding location in $t.$ Occluded points at $t-1,$ do not have a correspondence at $t,$ and therefore an ideal optical flow should not be defined at these locations. The same applies for the backward optical flow.

In practice, most optical flow algorithms compute a dense correspondence from one frame to another. Thus we have a forward dense mapping from $t-1$ to $t$ given by the forward optical flow as $\varvec{\varphi }^f_{t-1}:\varOmega _{t-1}\rightarrow \varOmega _t,$ as $\varvec{\varphi }^f_{t-1}(\varvec{x}) = (\varvec{x} + \varvec{v}^f(\varvec{x},t-1),t).$ Similarly, we have a backward dense mapping at $t$ given by the backward optical flow, $\varvec{\varphi }^b_t:\varOmega _{t}\rightarrow \varOmega _{t-1}.$

We will focus in the following on the forward mapping. An analogous discussion holds for the backward mapping.

For a given $\varvec{x}\in \varOmega _t,$ we consider a neighborhood $Q_{\varvec{x}}\subset \varOmega _{t}$ given by $Q_{\varvec{x}} = \varvec{x} + (-1,1)^2.$ We denote by $[\varvec{\varphi }^f_{t-1}]^{-1}(Q_{\varvec{x}}) = \{\varvec{z}\in \varOmega _{t-1}:\varvec{\varphi }^f_{t-1}(\varvec{z})\in Q_{\varvec{x}}\}$ the pre-image of $Q_{\varvec{x}}$ under the forward mapping $\varphi ^f_{t-1}.$ Note that this set is also defined even when $\varvec{\varphi }^f_{t-1}$ is not invertible. We define the forward area as

$$\begin{aligned} \mathcal{A }^f_{t-1}(Q_{\varvec{x}}):= \int \limits _{[\varvec{\varphi }^f_{t-1}]^{-1}(Q_{\varvec{x}})}\text{ d}\varvec{z}. \end{aligned}$$

To compute this integral, we extend the discrete mapping $\varvec{\varphi }_{t-1}^f$ to a continuous spatial domain using a bi-linear interpolation.

If $Q_{\varvec{x}}$ is not being occluded nor dis-occluded from $t-1$ to $t,$ then $[\varvec{\varphi }^f_{t-1}]^{-1}$ is a well defined function on $Q_{\varvec{x}}.$ That is, every point in $Q_{\varvec{x}}$ has a unique pre-image by the forward mapping $\varvec{\varphi }^f_{t-1}.$ Furthermore, for a locally translational flow $\varvec{\varphi }^f_{t-1}$ we can expect $\mathcal{A }^f_{t-1}(Q_{\varvec{x}})$ to be close to the area of $Q_{\varvec{x}},$ given by $\mathcal{A }(Q_{\varvec{x}}) = \int _{Q_{\varvec{x}}}\text{ d}\varvec{z}.$ In the case of $Q_{\varvec{x}} = \varvec{x} + (-1,1)^2$ we have $\mathcal{A }(Q_{\varvec{x}}) = 4.$

Based on the comparison between $\mathcal{A }^f_{t-1}(Q_{\varvec{x}})$ and $\mathcal{A }(Q_{\varvec{x}})$ we will define potentially occluded regions to be further examined. We add a small margin of $\varepsilon =0.5$ to the comparison, to avoid marking all pixels as potentially occluded. We consider two cases:

• (1) $\mathcal{A }^f_{t-1}(Q_{\varvec{x}}) > \mathcal{A }(Q_{\varvec{x}}) + \varepsilon$: There is an excess of points mapped into $Q_{\varvec{x}}$ by $\varvec{\varphi }^f_{t-1}.$ This situation may arise when there is an occlusion from frame $t-1$ to $t,$ causing several points in $\varOmega _{t-1}$ to be mapped forward into the same location at frame $t.$ Some of them are being occluded, whereas others remain visible. Therefore, we mark all pixels in $[\varvec{\varphi }^f_{t-1}]^{-1}(Q_{\varvec{x}})$ as candidates for occluded regions, for further examination. Note that there may be other reasons for which $\mathcal{A }^f_{t-1}(Q_{\varvec{x}}) > \mathcal{A }(Q_{\varvec{x}}) + \varepsilon ,$ for example in the case of a zoom out. In this first step, we treat all of these cases equally.

• (2) $\mathcal{A }^f_{t-1}(Q_{\varvec{x}}) < \mathcal{A }(Q_{\varvec{x}})- \varepsilon$: Few points are mapped by $\varvec{\varphi }^f_{t-1}$ into $Q_{\varvec{x}}.$ This may occur if $Q_{\varvec{x}}$ lies in an region that has been dis-occluded from $t-1$ to $t.$ If this is the case, some points in $Q_{\varvec{x}}$ at frame $t,$ do not have a correspondence in $t-1,$ and we mark points in $Q_{\varvec{x}}$ as candidates for dis-occluded regions (or occluded when looking from $t$ to $t-1$). Their backward optical flow will be examined in the second step. Note that the forward flows arriving at the dis-occluded region might be as well wrong (no point in $t-1$ should have a correspondence in a dis-occluded region at $t$). Thus we also mark them as candidates for further examination.

The same process is then applied considering the backward area mapped into frame $t-1$ from $t$ by the backward optical flow.

Figure 25c shows an example output of the above discussed first-step occlusion detection.

Remark A more general method for detecting occlusion candidates can be derived based on the consistency between the forward and backward optical flows (Alvarez et al. 2007). In addition to the forward area $\mathcal{A }^f_{t-1}(Q_{\varvec{x}}),$ one could also define the backward area as

$$\begin{aligned} \mathcal{A }^b_t(Q_{\varvec{x}}):= \int \limits _{\varvec{\varphi }^b_{t}(Q_{\varvec{x}})}\!\!\!\!\!\text{ d}\varvec{z}. \end{aligned}$$

For an ideal optical flow, the forward and backward areas should be equal.

In an ideal case, the forward and backward flows are symmetrical (see for instance Alvarez et al. 2007). Thus, the area of the pre-image of $Q_{\varvec{x}}$ under the forward flow ($\mathcal{A }^f_{t-1}(Q_{\varvec{x}})$) and the area of the image of $Q_{\varvec{x}}$ under the backward flow ($\mathcal{A }^b_t(Q_{\varvec{x}})$) should be equal. Therefore, differences between these areas could be considered as evidence of occlusions or dis-occlusions. Cases 1 and 2 in the above discussion apply, using $\mathcal{A }^b_t(Q_{\varvec{x}})$ instead of $\mathcal{A }(Q_{\varvec{x}}).$ In practice, for the sequences we used, both approaches behave similarly.

1.2 1.2 Second Step, the Categorization into Occluded and Still Visible Regions

As a result of the first step, we have for each frame $t$ two sets $C_t^f,C_t^b\subset \varOmega _t$ of potentially occluded pixels. If a pixel $\varvec{x}$ belongs to $C_t^f,$ then its corresponding position at $\varOmega _{t+1},$ given by the forward optical flow $\varvec{\varphi }^f_t(\varvec{x}) = \varvec{x} + \varvec{v}^f(\varvec{x},t),$ might be occluded. Similarly, for a pixel $\varvec{x}\in C_t^b,$ its corresponding backward position $\varvec{\varphi }^b_t(\varvec{x}) = \varvec{x} + \varvec{v}^b(\varvec{x},t)$ in $\varOmega _{t-1}$ might be occluded. In the first step, all the analysis has been done based on the properties of the mapping between two frames, without considering the visibility of a pixel in the next frame. In this second step we try to determine which pixels in $C_t^f$ and $C_t^b$ are occluded and which are still visible. In what follows we describe this procedure for $C_t^f.$ The same applies to $C_t^b.$

We base our occlusion detection on the error between a patch from frame $t$ centered at $\varvec{x}\in C_t^f$ and its corresponding patch at frame $t+1,$ centered at $\varvec{x} + \varvec{v}^f(\varvec{x},t).$ We will denote by $p_u(\varvec{x},t)$ the patch at $(\varvec{x},t).$ We define the patch error as the squared $L_2$ distance between corresponding patches:

$$\begin{aligned} e^f(\varvec{x})&= \Vert p_u(\varvec{x},t) - p_u(\varvec{x} + \varvec{v}^f(\varvec{x},t),t+1)\Vert ^2 \nonumber \\&= \sum _{\varvec{h}\in \varOmega _p} (u(\varvec{x} + \varvec{h},t) \nonumber \\&\quad - u(\varvec{x} + \varvec{v}^f(\varvec{x},t) + \varvec{h},t+1))^2, \end{aligned}$$

(37)

where $\varOmega _p\subset \mathbb{Z }^2$ denotes the patch domain (a square neighborhood of $\varvec{0}\in \mathbb{Z }^2$ in our case).

To detect occluded regions we threshold $e^f.$ However, setting a constant threshold that is not input sensitive might lead to undesired results. For that, we set a dynamic threshold adapted to each image pair. Therefore, we pose the occlusion detection as the following statistical test:

•    $\mathcal{H }_0$:   the point at $(\varvec{x},t)$ is not occluded at $t+1$

•    $\mathcal{H }_1$:   the point at $(\varvec{x},t)$ is occluded at $t+1$   .

As the statistic for the test we consider the patch error $e^f,$ (we assume patch errors at different pixels to be independent). We estimate a statistical model for $\mathcal{H }_0$ by constructing a histogram of patch errors using the unlabeled points $S_t^f\setminus C_t^f.$

Based on this histogram, we compute a threshold $\tau ^f_t >0$ by fixing the false alarm rate $\alpha \in [0,1],$ i.e. , $\tau ^f_t$ such that $P(e^f > \tau ^f_t|\mathcal{H }_0) < \alpha .$ In this way we compute an adaptive threshold for each frame transition. Patch errors above the threshold are considered as occluded. Let us mention that this test is designed not to be tolerant to false negatives (i.e. , occluded pixels that have passed the test and been categorized as not occluded). Because of that, it is bound to allow for some false positives (i.e. , pixels that are not occluded being categorized as occluded). In practice this over-estimation does not affect the quality of the result.

After applying this second step, for each frame in the sequence, the occlusion detection scheme yields two sets $K^f_t,K^b_t\subset \varOmega _t$ of forward and backward occluded pixels.

Figure 25d shows an example output of the above discussed second-step occlusion detection.

2 The Euler-Lagrange Equation

Throughout the rest of the paper, we assume that $O$ is a subset of $\varOmega ^T=\varOmega \times [0,T]$ with Lipschitz boundary. Then the unit normal is defined almost everywhere on $\partial O$ with respect to the Hausdorff measure $\mathcal{H }^2$ (surface measure) on $\partial O.$ Let us denote by $\varvec{\nu }^O=(\varvec{\nu }^O_{\varvec{x}},{\nu }^O_t)$ the outer unit normal to $\partial O$ (a vector in the unit sphere of $\mathbb{R }^3$) and $\varvec{\nu }^{O_t}$ the outer unit normal to $\partial O_t$ (a vector in the unit circle of $\mathbb{R }^2$).

Let us compute the Euler-Lagrange equations associated to the energy

$$\begin{aligned} E_{{\varvec{\kappa }},\lambda }(u) = \int \limits _{O} \left(\frac{1}{2}\Vert {\varvec{\kappa }}(\varvec{x},t)\,\varvec{\nabla }\partial _{\varvec{v}} u(\varvec{x},t)\Vert ^2 + \frac{\lambda }{p} \Vert \varvec{\nabla } u(\varvec{x},t)\Vert ^p\right) \, \text{ d}\varvec{x}\,\text{ d}t, \end{aligned}$$

(38)

where $\lambda \ge 0$ and $p=1,2.$ For that, assume that $u:O\rightarrow \mathbb{R }$ is a minimum of $E_{{\varvec{\kappa }},\lambda }.$ To compute the Euler-Lagrange equations, we consider a perturbation $\bar{u}$ such that $E_{{\varvec{\kappa }},\lambda }(\bar{u}) < \infty .$ Since $u$ is a minimum of $E_{{\varvec{\kappa }},\lambda }$ we have

$$\begin{aligned} \lim _{\epsilon \rightarrow 0+} \frac{E_{{\varvec{\kappa }},\lambda }(u + \epsilon \bar{u})-E_{{\varvec{\kappa }},\lambda }(u)}{\epsilon }&= \int \limits _O {\varvec{\kappa }} \varvec{\nabla }\partial _{\varvec{v}} u \cdot {\varvec{\kappa }} \varvec{\nabla }\partial _{\varvec{v}} \bar{u} \, d\varvec{x} dt \\&+ \lambda \int \limits _O \varvec{\xi }\cdot \varvec{\nabla } \bar{u} \text{ d}\varvec{x}\,\text{ d}t = 0, \end{aligned}$$

where, when $\lambda > 0$ and $p=1,$ $\varvec{\xi }:O\rightarrow \mathbb{R }^2$ is a measurable vector field such that $\Vert \varvec{\xi }\Vert _\infty \le 1,$ $\varvec{\xi }\cdot \varvec{\nabla } u= \vert \varvec{\nabla } u \vert ,$ and the arguments $(\varvec{x},t)$ of the functions are omitted for simplicity. If $\lambda > 0$ and $p=2,$ then $\varvec{\xi } = \varvec{\nabla } u.$ Integrating by parts we have

$$\begin{aligned} 0&= \int \limits _O {\varvec{\kappa }} \varvec{\nabla }\partial _{\varvec{v}} u \cdot {\varvec{\kappa }} \varvec{\nabla }\partial _{\varvec{v}} \bar{u} \, d\varvec{x} \text{ d}t + \lambda \int \limits _O \varvec{\xi }\cdot \varvec{\nabla } \bar{u} \text{ d}\varvec{x}\,\text{ d}t\\&= \int \limits _O \partial _{\varvec{v}}^* \varvec{\nabla }^* ( {\varvec{\kappa }}{^2} \varvec{\nabla }\partial _{\varvec{v}} u) \bar{u} \, d\varvec{x} \text{ d}t + \lambda \int \limits _O \varvec{\nabla }^* \varvec{\xi }\, \bar{u} \text{ d}\varvec{x}\,\text{ d}t\\&+ \int \limits _{\partial O} \varvec{\nabla }^* ( {\varvec{\kappa }}{^2} \varvec{\nabla }\partial _{\varvec{v}} u) ({\nu }^O_t + \varvec{v}\cdot \varvec{\nu }^O_{\varvec{x}}) \bar{u} \,d\mathcal{H }^2 \nonumber \\&+ \lambda \int \limits _0^T \int \limits _{\partial O_t} \varvec{\xi }\cdot \varvec{\nu }^{O_t} \bar{u} d\mathcal{H }^1 \text{ d}t\\&+ \int \limits _0^T \int \limits _{\partial O_t} {\varvec{\kappa }}{^2} \varvec{\nabla }\partial _{\varvec{v}} u \cdot \varvec{\nu }^{O_t} \partial _{\varvec{v}} \bar{u} \, d\mathcal{H }^1 \text{ d}t, \end{aligned}$$

where $d\mathcal{H }^2,$ resp. $d\mathcal{H }^1,$ denotes the surface measure in $\partial O,$ resp. the length measure in $\partial O_t.$ We have denoted by $\varvec{\nabla }^*$ (resp. $\partial _{\varvec{v}}^*$) the adjoint operator, that is $\varvec{\nabla }^* b= - \text{ div} \, b$ for any vector field $b:O\rightarrow \mathbb{R }^2$ (resp. $\partial _{\varvec{v}}^* \psi = - \frac{\partial \psi }{\partial t} - \text{ div} \, (\varvec{v}\psi ),$ for any function $\psi :O\rightarrow \mathbb{R }$). By taking test functions that vanish in a neighborhood of the boundary we have $\bar{u} = 0,$ $\partial _{\varvec{v}} \bar{u} =0$ on $\partial O$ and we deduce that

$$\begin{aligned} \partial _{\varvec{v}}^* \varvec{\nabla }^* ( {\varvec{\kappa }}{^2} \varvec{\nabla }\partial _{\varvec{v}} u) + \lambda \varvec{\nabla }^* \varvec{\xi } = 0 \quad \text{ in}\quad O. \end{aligned}$$

Introducing this in the above expressions we get

$$\begin{aligned} \begin{array}{lll} \qquad \displaystyle \int \limits _{\partial O} \varvec{\nabla }^* ( {\varvec{\kappa }}{^2} \varvec{\nabla }\partial _{\varvec{v}} u) ({\nu }^O_t + \varvec{v}\cdot \varvec{\nu }^O_{\varvec{x}}) \bar{u} \, d\mathcal{H }^2 \\ \qquad \displaystyle + \lambda \int \limits _0^T \int \limits _{\partial O_t} \varvec{\xi }\cdot \varvec{\nu }^{O_t} \bar{u} d\mathcal{H }^1 \text{ d}t \\ \qquad \displaystyle + \int \limits _0^T \int \limits _{\partial O_t} {\varvec{\kappa }}{^2} \varvec{\nabla }\partial _{\varvec{v}} u \cdot \varvec{\nu }^{O_t} \partial _{\varvec{v}} \bar{u} \, d\mathcal{H }^1 \text{ d}t = 0 \end{array} \end{aligned}$$

(39)

and this holds for any admissible perturbation $\bar{u}$ that will be clarified below.

Let us discuss the boundary conditions that can be specified for the problem. We use the definition and notations given in Sect. 3.1. A set of natural boundary conditions are those for which the identity (39) holds. Let us discuss the possible choices.

1.1 2.1 Dirichlet Boundary Conditions

Dirichlet boundary conditions for $u$ can be specified on a given set $A \subset \partial O$ if $\lambda > 0$ or on a subset $A \subset \partial O \setminus \partial O_\mathrm{tang}$ if $\lambda = 0.$ Namely we can specify

$$\begin{aligned} u(\varvec{x},t) = u_0(\varvec{x},t) \quad (\varvec{x},t)\in A. \end{aligned}$$

(40)

If $u$ satisfies (40) and we take test functions $\bar{u}$ such that $\bar{u} = 0$ on $A,$ then $u+\epsilon \bar{u}$ satisfies (40) and the first and second integrals in (39) vanishes on $A.$

Observe that, since $\partial O$ is Lipschitz,

$$\begin{aligned} \{(\varvec{x},t): \varvec{x}\in \partial O_t, t\in (0,T)\} = \partial O_\mathrm{tang} \cup \partial O_\mathrm{obli}, \end{aligned}$$

where strictly speaking this equality holds modulo null sets with respect to the surface measure.

1.2 2.2 Specifying $\partial _{\varvec{v}} u$ on the Boundary

We can specify $\partial _{\varvec{v}} u$ on a given subset of $\{(\varvec{x},t): \varvec{x}\in \partial O_t, t\in (0,T)\}.$ Namely we can specify

$$\begin{aligned} \partial _{\varvec{v}} u(\varvec{x},t) = g_0(\varvec{x},t) \quad (\varvec{x},t)\in B \subset \partial O_\mathrm{tang} \cup \partial O_\mathrm{obli}. \end{aligned}$$

(41)

If $u$ satisfies (41) and we take test functions $\bar{u}$ such that $\partial _{\varvec{v}} \bar{u} = 0$ on $B\subset \partial O_\mathrm{tang} \cup \partial O_\mathrm{obli},$ then $u+\epsilon \bar{u}$ satisfies (41) and the third integral in (39) vanishes on $B.$

1.3 2.3 pecifying Other Boundary Conditions

We can specify the boundary condition at $(\varvec{x},t)\in A^\prime \subset \partial O$

$$\begin{aligned} \varvec{\nabla }^* ( {\varvec{\kappa }}{^2} \varvec{\nabla }\partial _{\varvec{v}} u) \varvec{\nu }^O \cdot (\varvec{v},1) + \lambda \varvec{\xi }\cdot \varvec{\nu }^{O_t}= 0 \end{aligned}$$

(42)

with the convention that $\varvec{\xi }\cdot \varvec{\nu }^{O_t}= 0$ if $(\varvec{x},t)\in \partial O_\mathrm{vert} \cup O_0\cup O_T.$ Then the sum of the first and second integrals in (39) vanishes on $A^\prime .$

Notice that if $\lambda = 0,$ (42) reduces to

$$\begin{aligned} \varvec{\nabla }^* ( {\varvec{\kappa }}^2 \varvec{\nabla }\partial _{\varvec{v}} u) \varvec{\nu }^O \cdot (\varvec{v},1) = 0 \end{aligned}$$

(43)

and is trivially satisfied if $(\varvec{x},t) \in \partial O_\mathrm{tang}$ since in that case $\varvec{\nu }^O \cdot (\varvec{v},1) = 0.$ That is, this gives no boundary condition at points $(\varvec{x},t) \in \partial O_\mathrm{tang}.$ Thus, when $\lambda = 0$ we can only impose (43) on subsets $A^{\prime }\subset \partial O \setminus \partial O_\mathrm{tang}.$

If $\lambda > 0,$ we can impose (42) on any subset ${A^{\prime }} \subset \partial O,$ understanding that it reduces to

$$\begin{aligned} \varvec{\xi }\cdot \varvec{\nu }^{O_t}= 0. \end{aligned}$$

(44)

1.4 2.4 Specifying ${\varvec{\kappa }}{^2} \varvec{\nabla }\partial _{\varvec{v}} u \cdot \varvec{\nu }^{O_t} = 0$ on the boundary

We can specify the boundary condition at $(\varvec{x},t)\in B^\prime \subset \partial O_\mathrm{tang} \cup \partial O_\mathrm{obli}$

$$\begin{aligned} {\varvec{\kappa }}{^2} \varvec{\nabla }\partial _{\varvec{v}} u \cdot \varvec{\nu }^{O_t} = 0. \end{aligned}$$

Then the second integral in (39) vanishes on $B^\prime .$

Depending on the problem we choose a set of boundary conditions. The only requirements are that

$$\begin{aligned} A \cup A^\prime = \partial O \quad \text{ if}\quad \lambda > 0,\quad \text{ or} \,\, A \cup A^\prime = \partial O \setminus \partial O_\mathrm{tang}\\&\text{ if} \lambda = 0, \end{aligned}$$

and

$$\begin{aligned} B \cup B^\prime = \partial O_\mathrm{tang} \cup \partial O_\mathrm{obli}. \end{aligned}$$

This implies that the identity (39) holds.

1.5 2.5 Boundary Conditions for the One-Lid Setting

In the context of the one-lid problem, we choose the set of boundary conditions

$$\begin{aligned}&u(\varvec{x},0) = u_0(\varvec{x},0),&\varvec{x}&\in O_0, \end{aligned}$$

(45)

$$\begin{aligned}&u(\varvec{x},t) = u_0(\varvec{x},t),&(\varvec{x},t)&\in \partial O_{\mathrm{vert}}, \end{aligned}$$

(46)

$$\begin{aligned}&\!\begin{array}{l} \partial _{\varvec{v}} u(\varvec{x},t) = g_0(\varvec{x},t) \end{array},&(\varvec{x},t)&\in \partial O_{\mathrm{tang}} \setminus \partial \varOmega ^T, \end{aligned}$$

(47)

$$\begin{aligned}&\!\begin{array}{l} u(\varvec{x},t) = u_0(\varvec{x},t) \\ \partial _{\varvec{v}} u(\varvec{x},t) = g_0(\varvec{x},t) \end{array},&(\varvec{x},t)&\in \partial O_{\mathrm{obli}} \setminus \partial \varOmega ^T, \end{aligned}$$

(48)

to which, when $\lambda > 0,$ we add

$$\begin{aligned} u(\varvec{x},t) = u_0(\varvec{x},t) \qquad \qquad \,\,\, \, (\varvec{x},t) \in \partial O_{\mathrm{tang}} \setminus \partial \varOmega ^T, \end{aligned}$$

(49)

where the videos $u_0$ and $g_0$ are given. Notice that the boundary condition (49) is interpreted classically if $p=2$ and it has to be interpreted in a relaxed sense if $p=1.$ This is discussed with more detail in Sadek et al. (2012).

The boundary conditions on the rest of $\partial O$ are

$$\begin{aligned}&\varvec{\nabla }^* ({\varvec{\kappa }}{^2} \varvec{\nabla } \partial _{\varvec{v}} u) (\varvec{x},t) = 0,&\varvec{x}&\in O_T, \end{aligned}$$

(50)

$$\begin{aligned}&\!\begin{array}{l} \lambda \varvec{\xi }\cdot \varvec{\nu }^{O_t}(\varvec{x},t) = 0\\ {\varvec{\kappa }}{^2} \varvec{\nabla } \partial _{\varvec{v}} u(\varvec{x},t)\cdot \varvec{\nu }^{O_t}(\varvec{x},t) = 0 \end{array},&(\varvec{x},t)&\in \partial O_{\mathrm{tang}}\cap \partial \varOmega ^T, \end{aligned}$$

(51)

$$\begin{aligned}&\!\begin{array}{l} \varvec{\nabla }^*({\varvec{\kappa }}{^2} \varvec{\nabla } \partial _{\varvec{v}} u) (\varvec{x},t) + \lambda \varvec{\xi }\cdot \varvec{\nu }^{O_t} (\varvec{x},t) =0\\ {\varvec{\kappa }}{^2} \varvec{\nabla } \partial _{\varvec{v}} u(\varvec{x},t)\cdot \varvec{\nu }^{O_t}(\varvec{x},t) = 0 \end{array}\!, \!\!\!\!\!\!&(\varvec{x},t)&\in \partial O_{\mathrm{obli}}\cap \partial \varOmega ^T. \end{aligned}$$

(52)

1.6 2.6 Boundary Conditions for the Two-Lid Setting

They are given by (45),(46),(47),(48),(51),(52), and (50) is replaced by

$$\begin{aligned} u(\varvec{x},T) = u_0(\varvec{x},T) \quad \text{ in}\quad O_T. \end{aligned}$$

(53)

Let us observe that the boundary conditions (45),(46),(47), (48), and (53) in the two lid-case, are specified in the set of admissible functions in which $E_{{\varvec{\kappa }},\lambda }$ will be minimized.

Remark

Under some assumptions on the vector field v, we can prove existence and uniqueness of minima of $E_{{\varvec{\kappa }},\lambda }$ in a suitable class of functions (the functional space where the energy is finite and permits to incorporate boundary conditions). In particular, this shows that the boundary conditions are sufficient to determine the solution. This is discussed in detail in Sadek et al. (2012).

 

3 Some Examples of Analytic Solutions of the Euler-Lagrange Equation

To illustrate how the prescribed boundary conditions determine the solution, we compute in this section the analytic solution of the Euler-Lagrange equation of the continuous energy $E_{{{\kappa }},\lambda }$ with $\lambda = 0.$ We consider two simple examples, one for the one-lid setting and one for the two-lid setting.

We will consider a simple case, in which $v(x,t) = 0$ everywhere in $\varOmega .$ In this case the convective derivative coincides with the partial derivative with respect to time: $\partial _{{v}} u = u_t.$ The energy is therefore

$$\begin{aligned} E(u) = \int \limits _O (u_{xt}(x,t))^2\text{ d}x\text{ d}t. \end{aligned}$$

(54)

We consider an editing domain $O = \{(x,t): t\in [0,T], x\in [a(t),b(t)]\},$ such as the one depicted in Fig. 26. We consider $a:[0,T]\rightarrow \mathbb{R }$ and $b:[0,T]\rightarrow \mathbb{R }$ to be functions with a continuous bounded derivative, and such that $a(t) < b(t)$ for $t\in [0,T].$ We also suppose for simplicity that $a$ is a strictly decreasing function, whereas $b$ is strictly increasing.

Fig. 26

Domain and boundary conditions for a one-lid problem. The optical flow in this example is zero

1.1 3.1 One-Lid Problem

For the one-lid problem, the boundary conditions are as follows:

$$\begin{aligned}&u(x,0) = u_0(x),&x\in [a(0),b(0)], \end{aligned}$$

(55)

$$\begin{aligned}&u(a(t),t) = u_a(t),&t\in [0,T], \end{aligned}$$

(56)

$$\begin{aligned}&u(b(t),t) = u_b(t),&t\in [0,T], \end{aligned}$$

(57)

$$\begin{aligned}&u_t(a(t),t) = g_a(t),&t\in [0,T], \end{aligned}$$

(58)

$$\begin{aligned}&u_t(b(t),t) = g_b(t),&t\in [0,T]. \end{aligned}$$

(59)

The minimum of the energy can be computed by solving the Euler-Lagrange equation:

$$\begin{aligned} u_{txxt}(x,t) = 0, \quad \quad \forall (x,t)\in O \end{aligned}$$

(60)

with the following additional boundary condition, stemming from the computation of the first variation:

$$\begin{aligned} u_{xxt}(x,T) = 0,\quad \quad \quad \quad \quad x\in [a(T),b(T)]. \end{aligned}$$

(61)

Let us compute the solution of the PDE (60). Integrating it with respect to $t,$ between $t$ and $T$ yields,

$$\begin{aligned} u_{xxt}(x,t) = u_{xxt}(x,T) = 0, \quad \quad \forall (x,t)\in O, \end{aligned}$$

the last equality is due to the Neumann boundary condition at $t = T.$ If we now integrate w.r.t. $x,$ between $a(t)$ and $x$ we obtain

$$\begin{aligned} u_{xt}(x,t) = u_{xt}(a(t),t), \quad \quad \forall (x,t)\in O. \end{aligned}$$

We integrate again on variable $x$:

$$\begin{aligned}&u_t(x,t) - u_t(a(t),t) \nonumber \\&\quad = \int \limits _{a(t)}^x u_{xt}(a(t),t)\text{ d}s = u_{xt}(a(t),t)(x - a(t)). \end{aligned}$$

If we evaluate this expression on ${x} = b(t),$ we can express $u_{xt}(a(t),t)$ in terms of $g_a$ and $g_b$:

$$\begin{aligned} u_{xt}(a(t),t) = \frac{u_t(b(t),t) - u_t(a(t),t)}{b(t)- a(t)}= \frac{g_b(t) - g_a(t)}{b(t)- a(t)}. \end{aligned}$$

Therefore we have that

$$\begin{aligned} u_t(x,t) = g_a(t) + \frac{g_b(t) - g_a(t)}{b(t)- a(t)}(x - a(t)). \end{aligned}$$

(62)

As a function of $x,$ $u_t(x,t)$ is a linear function passing through $(a(t),g_a(t))$ and $(b(t),g_b(t)),$ i.e. , the rate of illumination change is a smooth interpolation of the values specified at the boundary.

To obtain the solution of the PDE we now integrate with respect to $t.$ Let us define the function $\ell :[a(T),b(T)]\rightarrow R$ as

$$\begin{aligned}\ell (x) = \left\{ \begin{array}{l l} a^{-1}(x)&\text{ if} a(T) \leqslant x \leqslant a(0),\\ 0&\text{ if} a(0) < x < b(0),\\ b^{-1}(x)&\text{ if} b(0) \leqslant x \leqslant b(T).\\ \end{array} \right. \end{aligned}$$

Note that $(x,\ell (x))$ with ${x}\in [a(T),b(T)],$ corresponds to the “left” boundary of $O,$ where the value of $u$ is specified by $u_a,$ $u_0$ and $u_b.$ We now integrate (62) on $t$ between $\ell (x)$ and $t,$ yielding

$$\begin{aligned}&u(x,t) - u_0(x,\ell (x)) \nonumber \\&\quad = \int \limits _{\ell (x)}^t \!\!\!\!g_a(s)\text{ d}s +\! \int \limits _{\ell (x)}^t \!\!\!\!\frac{g_b(s) - g_a(s)}{b(s)- a(s)}(x - a(s))\text{ d}s. \end{aligned}$$

This example demonstrates that the given boundary conditions are sufficient to compute a minimizer of the energy. It also shows how each boundary condition is used.

In this example, for fixed $t,$ the illumination change rate $\partial _v u(x,t) = u_t(x,t)$ is the result of a linear interpolation between the illumination change rate given at the boundary, $g_a(t)$ and $g_b(t).$ Once the illumination change rate is known in the whole editing domain, the solution is computed by integrating it along the trajectories. The temporal integration starts at $\ell (x),$ the time instant where the trajectory through $(x,t)$ reaches a point of the boundary where $u_0$ is specified (including the first lid where $\ell (x) = 0$). This integration starts with $u_0(x,\ell (x))$ and propagates it along the trajectory, while accommodating for the illumination changes previously computed in $u_t.$

1.2 3.2 Two-Lid Problem

For the two-lid problem, in addition to the boundary conditions of the one-lid problem, we add the second lid:

$$\begin{aligned} u(x,T) = u_T(x), \quad x\in [a(T),b(T)] \end{aligned}$$

(63)

in substitution of (61). We use the notation of Sect. .

Let us compute the solution of

$$\begin{aligned} u_{txxt}(x,t) = 0, \quad \quad (x,t)\in O. \end{aligned}$$

(64)

Observe that

$$\begin{aligned} O = \{(x,t)\in \varOmega \times [0,T]: t \ge \ell (x)\, \, x\in [a(t),b(t)]\}. \end{aligned}$$

Integrating with respect to $t,$ we have

$$\begin{aligned} u_{xxt}(x,t) = u_{xxt}(x,\ell (x)) \quad (x,t)\in O. \end{aligned}$$

(65)

Let us denote $G(x) = u_{xxt}(x,\ell (x)).$ Integrating again with respect to $t,$ we get

$$\begin{aligned} u_{xx}(x,t) = u_{xx}(x,\ell (x)) + G(x) (t-\ell (x)), \quad (x,t)\in O.\nonumber \\ \end{aligned}$$

(66)

For $t=T$ and $x\in [a(T),b(T)]$ we have

$$\begin{aligned} u_{xx}(x,T) = u_{xx}(x,\ell (x)) + G(x) (T-\ell (x)). \end{aligned}$$

(67)

Evaluating the above expression for $x\in [a(0),b(0)],$ we obtain

$$\begin{aligned} G(x)= \frac{1}{T} (u_{xx}(x,T)-u_{xx}(x,0)) \quad x\in [a(0),b(0)]. \end{aligned}$$

(68)

Integrating (66) with respect to $x,$ we obtain

$$\begin{aligned} u_x(x,t)&= u_x(a(t),t) + \int \limits _{a(t)}^x u_{xx}(s,\ell (s))\, \text{ d}s \nonumber \\&+ \int \limits _{a(t)}^x G(s)(t-\ell (s))\, \text{ d}s. \end{aligned}$$

(69)

Integrating (69) with respect to $x$ from $a(t)$ to $x,$ we obtain

$$\begin{aligned} u(x,t)&= u(a(t),t) + u_x (a(t),t)(x-a(t)) \nonumber \\&+ \int \limits _{a(t)}^x \int \limits _{a(t)}^y u_{xx}(s,\ell (s))\, \text{ d}s\, \text{ d}y\nonumber \\&+ \int \limits _{a(t)}^x \int \limits _{a(t)}^y G(s)(t-\ell (s))\, \text{ d}s\, \text{ d}y \end{aligned}$$

(70)

and integrating (69) from $x$ to $b(t),$ we obtain

$$\begin{aligned} u(x,t)&= u(b(t),t) + u_x (a(t),t)(x-b(t))\nonumber \\&- \int \limits _x^{b(t)} \int \limits _{a(t)}^y u_{xx}(s,\ell (s))\, \text{ d}s\, \text{ d}y \nonumber \\&- \int \limits _x^{b(t)} \int \limits _{a(t)}^y G(s)(t-\ell (s))\, \text{ d}s\, \text{ d}y. \end{aligned}$$

(71)

Evaluating (70) for $x=b(t)$ we obtain

$$\begin{aligned} u_b(t)&= u_a(t) + u_x (a(t),t)(b(t)-a(t)) \nonumber \\&+ \int \limits _{a(t)}^{b(t)} \int \limits _{a(t)}^y u_{xx}(s,\ell (s))\, \text{ d}s\, \text{ d}y\nonumber \\&+ \int \limits _{a(t)}^{b(t)} \int \limits _{a(t)}^y G(s)(t-\ell (s))\, \text{ d}s\, \text{ d}y. \end{aligned}$$

(72)

Using (67) we obtain

$$\begin{aligned} \begin{array}{ll} \displaystyle (t-T)\int \limits _{a(t)}^{b(t)} \int \limits _{a(t)}^y G(s)\, \text{ d}s\, \text{ d}y = A(t), \end{array} \end{aligned}$$

(73)

where

$$\begin{aligned} \begin{array}{ll} \displaystyle A(t):= u_b(t) - u_a(t) - u_x (a(t),t)(b(t)-a(t)) \\ \\ \displaystyle \qquad \qquad - \int \limits _{a(t)}^{b(t)} \int \limits _{a(t)}^y u_{xx}(s,T)\, \text{ d}s\, \text{ d}y. \end{array} \end{aligned}$$

(74)

Notice that by differentiating (56) with respect to $t$ and using (58) we obtain

$$\begin{aligned} u_x (a(t),t) = \frac{1}{a^{\prime }(t)} (u_a^\prime (t)-g_a(t)). \end{aligned}$$

(75)

Similarly, we obtain

$$\begin{aligned} u_x (b(t),t) = \frac{1}{{b^{\prime }}(t)} (u_b^\prime (t)-g_b(t)). \end{aligned}$$

(76)

Now, taking derivatives with respect to $t$ in (71) and evaluation the expression for $x=b(t)$ we obtain

$$\begin{aligned}&\!\!\!(u_x (a(t),t){-}u_x (b(t),t) {-} u_x (a(t),T){+}u_x (b(t),T))b^\prime (t)\nonumber \\&= -\int \limits _{a(t)}^{b(t)} G(s)(T-\ell (s)) \, \text{ d}s - \int \limits _{a(t)}^{b(t)} G(s)(t-\ell (s)) \, \text{ d}s.\nonumber \\ \end{aligned}$$

(77)

Let $B(t)$ be the left hand side of (77). By differentiating twice (73) with respect to $t,$ we get

$$\begin{aligned} G(b(t))b^\prime (t) - G(a(t))a^\prime (t) = Q^\prime (t), \end{aligned}$$

(78)

where

$$\begin{aligned} Q(t)= \frac{1}{(t-T)b^\prime (t)} \left( A^\prime (t)-\frac{A(t)}{(t-T)}\right) . \end{aligned}$$

By differentiating (77) with respect to $t,$ we get

$$\begin{aligned} G(b(t))b^\prime (t) - G(a(t))a^\prime (t) = \frac{B^\prime (t)+ Q(t)}{t-T}. \end{aligned}$$

(79)

From (78), (79) we get $G(a(t))$ and $G(b(t)).$ Together with $G(x),$ $x\in [a(0),b(0)]$ we have $G(x)$ for all $x\in [a(T),b(T)].$

From (67) we obtain $u_{xx}(x,\ell (x)).$ Then we have all ingredient to evaluate $u(x,t)$ using (70).

This example illustrates in a simple case how the boundary conditions are used to determine the solution in the two-lid setting.