Variational Texture Synthesis with Sparsity and Spectrum Constraints

Abstract

This paper introduces a new approach for texture synthesis. We propose a unified framework that both imposes first order statistical constraints on the use of atoms from an adaptive dictionary, as well as second order constraints on pixel values. This is achieved thanks to a variational approach, the minimization of which yields local extrema, each one being a possible texture synthesis. On the one hand, the adaptive dictionary is created using a sparse image representation rationale, and a global constraint is imposed on the maximal number of use of each atom from this dictionary. On the other hand, a constraint on second order pixel statistics is achieved through the power spectrum of images. An advantage of the proposed method is its ability to truly synthesize textures, without verbatim copy of small pieces from the exemplar. In an extensive experimental section, we show that the resulting synthesis achieves state of the art results, both for structured and small scale textures.

Appendix 1: Proofs

1.1 Projection on ${\fancyscript{C}_{\mathrm {s}}}$

We prove expression (8) of the projection of $u$ on ${\fancyscript{C}_{\mathrm {s}}}$. The projection is $u_s\in {\fancyscript{C}_{\mathrm {s}}}$ of the form $\hat{u}_s(m) = e^{\mathrm {i}\varphi (m)} \hat{u}_0(m)$. Our goal is to minimize

$$\begin{aligned} \left\| u-u_s\right\| ^2 = \sum _m \left\| \hat{u}(m) - e^{\mathrm {i}\varphi (m)}\hat{u}_0(m)\right\| ^2 \end{aligned}$$

(28)

with respect to $\varphi$.

As the term of the sum are independent, the problem is to minimize

$$\begin{aligned} f(\psi ) = \left\| x-ye^{\mathrm {i}\psi }\right\| ^2 \end{aligned}$$

(29)

where $x=\hat{u}(m)$, $y=\hat{u}_0(m)$ and $\psi =\varphi (m)$ for any $m$. The hermitian product of $x,y \in \mathbb {C}^3$ is denoted by $x\cdot y = \sum _ix_iy_i^* \in \mathbb {C}$

The development of the expression of $f(\psi )$ gives

$$\begin{aligned} f(\psi ) = \left\| x\right\| ^2 - e^{\mathrm {i}\psi } y\cdot x - e^{-\mathrm {i}\psi } x\cdot y + \left\| y\right\| ^2. \end{aligned}$$

The function $f$ being continuous and $2\pi$-periodic on $\mathbb {R}$, it admits (at least) a minimum and a maximum which are critical points $\psi _c$ satisfying $f'(\psi _c)=0$. Let’s write $x\cdot y=Ae^{\mathrm {i}\theta }$ with $A\ge 0$. The derivative

$$\begin{aligned} f'(\psi ) = - \mathrm {i}e^{\mathrm {i}\psi } y\cdot x + \mathrm {i}e^{-\mathrm {i}\psi } x\cdot y \end{aligned}$$

gives $e^{2\mathrm {i}\psi _c}=e^{2\mathrm {i}\theta }$ and the critical points $\psi _c$ are thus characterized by $e^{\mathrm {i}\psi _c}=\pm e^{\mathrm {i}\theta }$.

The second derivative

$$\begin{aligned} f''(\psi ) = e^{\mathrm {i}\psi } y\cdot x +e ^{-i\psi } x\cdot y. \end{aligned}$$

provides more information: we know $e^{\mathrm {i}\theta }$ is a minimum since $f''(e^{\mathrm {i}\theta })=2A\ge 0$, and $-e^{\mathrm {i}\theta }$ is a maximum since $f''(-e^{\mathrm {i}\theta })=-2A\le 0$.

The case $x\cdot y=0$ leads to $A=0$ and $f$ being constant. In other cases, $A>0$: the minimums $\psi _\mathrm {min}$ of the functions are strict and satisfy $e^{\mathrm {i}\psi _\mathrm {min}}=e^{\mathrm {i}\theta }=\frac{x\cdot y}{{\left| x\cdot y\right| }}$, hence the expression of $\hat{u}_s(m)$ given in (8).

1.2 Projection on ${\fancyscript{C}_{\mathrm {p}}}$

We provide here the proof that (15) and (16) are the minimizers of (12).

Using (17), the expression (12) is $\left\| R^{\left( \ell \right) }- w D_0 E_{n,k}\right\| ^2$. Expressing this norm as the sum of the norms of the columns leads to

$$\begin{aligned} \min _{k,n,w} \left\| R^{\left( \ell \right) }_k - w D_n\right\| ^2 + \sum _{k'\ne k} \left\| R^{\left( \ell \right) }_{k'}\right\| ^2\!. \end{aligned}$$

(30)

Let us fix $k$ and $n$. The problem

$$\begin{aligned} \min _w \left\| R^{\left( \ell \right) }_k-wD_n\right\| ^2 \end{aligned}$$

(31)

is now an orthogonal projection. Since $D_n$ is normalized, the solution is

$$\begin{aligned} w^* = \langle R^{\left( \ell \right) }_k,D_n \rangle \end{aligned}$$

(32)

as stated in (16) and Pythagora’s theorem gives the error

$$\begin{aligned} \left\| R^{\left( \ell \right) }_k - w^* D_n\right\| ^2 = \left\| R^{\left( \ell \right) }_k\right\| ^2 - \langle R^{\left( \ell \right) }_k,D_n \rangle ^2. \end{aligned}$$

(33)

Substituting $w$ in (30) by its optimal value $w^*$ function of $(k,n)$ simplifies the problem to

$$\begin{aligned} \min _{k,n} - \langle R^{\left( \ell \right) }_k,D_n \rangle ^2 + \sum _{k'} \left\| R^{\left( \ell \right) }_{k'}\right\| ^2\!. \end{aligned}$$

(34)

Hence the result (17) stating $(k^*,n^*) = {{\mathrm{arg\,max}}}{\left| \langle R^{\left( \ell \right) }_k,D_n \rangle \right| }$ among the set $\fancyscript{I}_{W^{\left( \ell \right) }}$ of admissible $(k,n)$.

1.3 Complexity of the greedy Algorithm 1

Here we show that the complexity of the greedy Algorithm 1 without the back-projection step is

$$\begin{aligned} \fancyscript{O}\big (KN(L+\lambda S\log K)\big ). \end{aligned}$$

(35)

Initialization. Computing the inner products $\varPhi =D_0^T P$ between the $K$ patches and the $N$ atoms in dimension $L$ requires $\fancyscript{O}(KNL)$ operations using standard matrix multiplications. Precomputing the inner products $D_0^T D_0$ is in $\fancyscript{O}(N^2 L)$ operations. Initializing $W$ and $R$ is in $KN$ operation.

Building a max-heap from the absolute values of the $KN$ inner products $\varPhi _{n,k} = \langle R_k,D_n \rangle$ has a time-complexity of $\fancyscript{O}\big (KN \log KN))$.

Loop for $\ell =1$ to $\lambda SK$. Finding the indices $(k^*,n^*)$ and extracting the maximum ${\left| \varPhi _{n^*,k^*}\right| }$ from the heap requires $\fancyscript{O}(\log KN)$ operations.

Updating $W$ with the optimal weight $w^*=\varPhi _{n^*,k^*}$ is only one operation. Updating $R$ is in $\fancyscript{O}(L)$ operations since only the $L$ coefficients of the $k^*$-th column is affected. Similarly, updating $\varPhi$ is in $\fancyscript{O}(N \log KN)$ operations since the $N$ updated coefficients must be relocated in the heap, and we recall that $D_0^T D_0$ is precomputed.

Conclusion. Since we assume that $S\ll L\le N\ll K$, previous bounds can be simplified, and in particular $\log (KN)\in \fancyscript{O}(\log K)$. The initializations and precomputations are in $\fancyscript{O}(KNL)$. Updating the heap is in $\fancyscript{O}(\lambda SKN \log K)$; building and searching the heap are cheaper. Hence the complexity (35).