Transport Between RGB Images Motivated by Dynamic Optimal Transport

Abstract

We propose two models for the interpolation between RGB images based on the dynamic optimal transport model of Benamou and Brenier (Numer Math 84:375–393, 2000). While the application of dynamic optimal transport and its extensions to unbalanced transform were examined for gray-value images in various papers, this is the first attempt to generalize the idea to color images. The non-trivial task to incorporate color into the model is tackled by considering RGB images as three-dimensional arrays, where the transport in the RGB direction is performed in a periodic way. Following the approach of Papadakis et al. (SIAM J Imaging Sci 7:212–238, 2014) for gray-value images we propose two discrete variational models, a constrained and a penalized one which can also handle unbalanced transport. We show that a minimizer of our discrete model exists, but it is not unique for some special initial/final images. For minimizing the resulting functionals we apply a primal-dual algorithm. One step of this algorithm requires the solution of a four-dimensional discretized Poisson equation with various boundary conditions in each dimension. For instance, for the penalized approach we have simultaneously zero, mirror, and periodic boundary conditions. The solution can be computed efficiently using fast Sin-I, Cos-II, and Fourier transforms. Numerical examples demonstrate the meaningfulness of our model.

Appendices

Appendix 1: Diagonalization of Structured Matrices

In the following we collect known facts on the eigenvalue decomposition of various difference matrices. For further information we refer, e.g., to [39, 46]. The following matrices $F_n$, $C_n$ and $S_n$ are unitary, resp., orthogonal matrices. The Fourier matrix

$$\begin{aligned} F_n := \, \sqrt{\tfrac{1}{n}} \left( \mathrm {e}^{\frac{-2\pi \mathrm {i}jk}{n}} \right) _{j,k=0}^n \end{aligned}$$

diagonalizes circulant matrices, i.e., for $a := (a_j)_{j=0}^{n-1} \in \mathbb R^n$ we have

$$\begin{aligned} { \left( \begin{array}{llll} a_0&{}a_{n-1}&{} \ldots &{} a_1\\ a_1&{}a_0&{}\ldots &{}a_2\\ \vdots &{} &{} \ddots &{} \vdots \\ a_{n-1} &{}a_1&{}\ldots &{}a_0 \end{array} \right) }&= \bar{F}_n \mathrm{diag}(\sqrt{n} F_n a) F_n\nonumber \\&= F_n \mathrm{diag}(\sqrt{n} \, \bar{F}_n a) \bar{F}_n. \end{aligned}$$

(21)

In particular it holds

$$\begin{aligned} \varDelta _n^{\mathrm{per}}&\,:=\,\frac{1}{n^2} (D_n^{\mathrm{per}})^{\scriptscriptstyle {\text {T}}}D_n^{\mathrm{per}} = \frac{1}{n^2} D_n^{\mathrm{per}} (D_n^{\mathrm{per}})^{\scriptscriptstyle {\text {T}}}\nonumber \\&= {\left( \begin{array}{rrrrrrrr} 2 &{}-1&{} &{} &{} &{} &{} &{}-1 \\ -1&{} 2&{}-1&{} \\ &{} &{} &{}\ddots &{} &{} \\ &{} &{} &{} &{} &{}-1&{}2 &{}-1 \\ -1&{} &{} &{} &{} &{} &{}-1&{} 2 \end{array} \right) }= \bar{F}_n \mathrm{diag}(\mathrm{d}^{\mathrm{per}}_n ) F_n \end{aligned}$$

(22)

with $\mathrm{d}^{\mathrm{per}}_n := \left( 4 \sin ^2 \frac{k\pi }{n}\right) _{k=0}^{n-1}$. The operator $\varDelta _n^{\mathrm{per}}$ typically appears when solving the one-dimensional Poisson equation with periodic boundary conditions by finite difference methods.

The DST-I matrix

$$\begin{aligned} S_{n-1} \,:=\, \sqrt{\tfrac{2}{n}} \left( \sin \frac{jk\pi }{n} \right) _{j,k=1}^{n-1}, \end{aligned}$$

and the DCT-II matrix

$$\begin{aligned} C_n := \sqrt{\tfrac{2}{n}} \left( \epsilon _j \cos \frac{j(2k+1)\pi }{2n} \right) _{j,k=0}^{n-1} \end{aligned}$$

with $\epsilon _0 := 1/\sqrt{2}$ and $\epsilon _j := 1$, $j=1,\ldots ,n-1$ are related by

$$\begin{aligned} D_n = S_{n-1} \left( 0 \, | \, \mathrm{diag}(\mathrm{d}^{\mathrm{zero}}_{n-1})^\frac{1}{2} \right) C_n, \end{aligned}$$

(23)

where $\mathrm{d}^{\mathrm{zero}}_{n-1} := \left( 4 \sin ^2 \frac{k\pi }{2n}\right) _{k=1}^{n-1}$. Further they diagonalize sums of certain symmetric Toeplitz and persymmetric Hankel matrices. In particular it holds

$$\begin{aligned} \varDelta _{n-1}^{\mathrm{zero}} \,&:=\, \frac{1}{n^2} D_{n} D_{n} ^{\scriptscriptstyle {\text {T}}}\nonumber \\&= {\left( \begin{array}{rrrrrrrr} 2 &{}-1&{} &{} &{} &{} &{} &{}0 \\ -1&{} 2&{}-1&{} \\ &{} &{} &{}\ddots &{} &{} \\ &{} &{} &{} &{} &{}-1&{}2 &{}-1 \\ 0&{} &{} &{} &{} &{} &{}-1&{} 2 \end{array} \right) }\nonumber \\&= S_{n-1} \mathrm{diag}(\mathrm{d}^{\mathrm{zero}}_{n-1}) S_{n-1} \end{aligned}$$

(24)

and

$$\begin{aligned} \varDelta _n^{\mathrm{mirr}}&:=\, \frac{1}{n^2} D_{n}^{\scriptscriptstyle {\text {T}}}D_{n} \nonumber \\&= {\left( \begin{array}{rrrrrrrr} 1 &{}-1&{} &{} &{} &{} &{} &{}0 \\ -1&{} 2&{}-1&{} \\ &{} &{} &{}\ddots &{} &{} \\ &{} &{} &{} &{} &{}-1&{}2 &{}-1 \\ 0&{} &{} &{} &{} &{} &{}-1&{} 1 \end{array} \right) } =C_n^{\scriptscriptstyle {\text {T}}}\mathrm{diag}(\mathrm{d}^{\mathrm{mirr}}_n) C_n \end{aligned}$$

(25)

with $\mathrm{d}^{\mathrm{mirr}}_n := \begin{pmatrix} 0\\ \mathrm{d}_{n-1}^{\mathrm{zero}} \end{pmatrix} = \left( 4\sin ^2 \frac{j \pi }{2n} \right) _{j=0}^{n-1}$. The operators $\varDelta _{n-1}^{\mathrm{zero}}$ and $\varDelta _{n}^{\mathrm{mirr}}$ are related to the Poisson equation with zero boundary conditions and mirror boundary conditions, respectively.

Appendix 2: Computation with Tensor Products

The tensor product (Kronecker product) of matrices

$$\begin{aligned} A&= \begin{pmatrix} a_{1,1} &{} \cdots &{} a_{1,n}\\ \vdots &{} \cdots &{} \vdots \\ a_{m,1} &{} \cdots &{} a_{m,n} \end{pmatrix} \in \mathbb {C}^{m, n} \end{aligned}$$

and

$$\begin{aligned} B&= \begin{pmatrix} b_{1,1} &{} \cdots &{} b_{1,t}\\ \vdots &{} \cdots &{} \vdots \\ b_{s,1} &{} \cdots &{} b_{s,t} \end{pmatrix} \in \mathbb {C}^{s, t} \end{aligned}$$

is defined by

$$\begin{aligned} A \otimes B := \begin{pmatrix} a_{1,1} B &{} \cdots &{} a_{1,n} B\\ \vdots &{} \ddots &{} \vdots \\ a_{m,1} B &{} \cdots &{} a_{m,n} B \end{pmatrix} \in \mathbb C^{ms, nt}. \end{aligned}$$

The tensor product is associative and distributive with respect to the addition of matrices.

Lemma 2

(Properties of Tensor Products)

• (i) $( A \otimes B)^{\scriptscriptstyle {\text {T}}}= A^{\scriptscriptstyle {\text {T}}}\otimes B^{\scriptscriptstyle {\text {T}}}$ for $A \in \mathbb {C}^{m, n}$, $B \in \mathbb {C}^{s , t}$. Let $A, C \in \mathbb {C}^{m, m}$ and $B, D \in \mathbb {C}^{n , n}$. Then the following holds:

• (ii) $( A \otimes B)( C \otimes D) = A C \otimes B D$ for $A, C \in \mathbb {C}^{m, m}$ and $B, D \in \mathbb {C}^{n , n}$.

• (iii) If A and B are invertible, then $A \otimes B$ is also invertible and

  $$\begin{aligned} ( A \otimes B)^{-1} = A^{-1} \otimes B^{-1} \, . \end{aligned}$$

The tensor product is needed to establish the connection between images and their vectorized versions, i.e., we consider images $F\in \mathbb {R}^{n_1\times n_2}$ columnwise reshaped as

$$\begin{aligned} f : = \text {vec}(F) \in \mathbb {R}^{n_1 n_2}. \end{aligned}$$

Then the following relation holds true:

$$\begin{aligned} \text {vec}(A F B^{\scriptscriptstyle {\text {T}}}) = (B \otimes A) f. \end{aligned}$$

(26)

Appendix 3: Proofs and Generalization of the Tensor Product Approach to 3D

Proof of Proposition 3

By definition of A and using (25), (22), we obtain for periodic boundary conditions

$$\begin{aligned} A A^{\scriptscriptstyle {\text {T}}}&= I_P \otimes \big (D_N^{\mathrm{per}} \big )^{\scriptscriptstyle {\text {T}}}D_N^{\mathrm{per}} + D_P^{\scriptscriptstyle {\text {T}}}D_P \otimes I_N\\&= I_P \otimes N^2 \varDelta _N^{\mathrm{per}} + P^2 \varDelta _P^{\mathrm{mirr}} \otimes I_N\\&= \Big (C_P^{\scriptscriptstyle {\text {T}}}\otimes \bar{F}_N \Big ) \; \mathrm{diag}\Big ( I_P \otimes N^2 \mathrm{d}_N^{\mathrm{per}} \\&\ \ \ \ + P^2 \mathrm{d}_P^{\mathrm{mirr}} \otimes I_N \Big ) \; \Big (C_P \otimes F_N\Big ). \end{aligned}$$

Similarly we get with (25) for mirror boundary conditions

$$\begin{aligned} A A^{\scriptscriptstyle {\text {T}}}&= I_P \otimes D_{N-1}^{\scriptscriptstyle {\text {T}}}D_{N-1} + D_{P}^{\scriptscriptstyle {\text {T}}}D_{P} \otimes I_{N-1} \\&= I_P \otimes N^2 \varDelta _{N-1}^{\mathrm{mirr}} + P^2 \varDelta _P^{\mathrm{mirr}} \otimes I_{N-1}\\&= \Big (C_P^{\scriptscriptstyle {\text {T}}}\otimes C_{N-1}^{\scriptscriptstyle {\text {T}}}\Big ) \; \mathrm{diag}\Big ( I_P \otimes N^2 \mathrm{d}_{N-1}^{\mathrm{mirr}}\\&\ \ \ \ + P^2 \mathrm{d}_{N-1}^{\mathrm{mirr}} \otimes I_{N-1}\Big ) \Big (C_P \otimes C_{N-1}\Big ) \end{aligned}$$

which finishes the proof. $\square$

Proof of Proposition 4

By definition of A we obtain

$$\begin{aligned} \lambda A^{\scriptscriptstyle {\text {T}}}A + \tfrac{1}{\tau } I= & {} \left( \begin{array}{ll} \lambda D_\mathrm{m}^{\scriptscriptstyle {\text {T}}}D_\mathrm{m} + \tfrac{1}{\tau } I&{} \quad \lambda D_\mathrm{m}^{\scriptscriptstyle {\text {T}}}D_\mathrm{f}\\ \lambda D_\mathrm{f}^{\scriptscriptstyle {\text {T}}}D_\mathrm{m}&{}\quad \lambda D_\mathrm{f}^{\scriptscriptstyle {\text {T}}}D_\mathrm{f} + \frac{1}{\tau } I \end{array} \right) \\= & {} \!\!{:} \left( \begin{array}{cc} X &{} Y\\ Y^{\scriptscriptstyle {\text {T}}}&{} Z \end{array} \right) \end{aligned}$$

so that the inverse can be written by the help of the Schur complement

$$\begin{aligned} S := Z - Y^{\scriptscriptstyle {\text {T}}}X^{-1} Y \end{aligned}$$

as

$$\begin{aligned} \left( \begin{array}{ll} X &{} Y\\ Y^{\scriptscriptstyle {\text {T}}}&{} Z \end{array} \right) ^{-1} = \left( \begin{array}{ll} I &{} - X^{-1}Y\\ 0 &{} I \end{array} \right) \left( \begin{array}{ll} X^{-1} &{} 0\\ 0 &{} S^{-1} \end{array} \right) \left( \begin{array}{ll} I &{} 0\\ -Y^{\scriptscriptstyle {\text {T}}}X^{-1}&{} I \end{array} \right) . \end{aligned}$$

By (22) and (24) we have with $D \in \{D_N^{\mathrm{per}},D_N\}$ that

$$\begin{aligned} X^{-1}&= \left( \lambda D_\mathrm{m}^{\scriptscriptstyle {\text {T}}}D_\mathrm{m} + \tfrac{1}{\tau } I\right) ^{-1} = \left( I_P \otimes \lambda N^2 D D^{\scriptscriptstyle {\text {T}}}+\tfrac{1}{\tau } I\right) ^{-1} \\&= I_P \otimes (\lambda N^2 D D^{\scriptscriptstyle {\text {T}}}+\tfrac{1}{\tau } I)^{-1} \\&= {\left\{ \begin{array}{ll} I_P \otimes S_{N-1} \mathrm{diag}(\lambda N^2 \mathrm{d}_{N-1}^{\mathrm{zero}} + \tfrac{1}{\tau }) ^{-1} S_{N-1} &{} \\ \qquad \qquad \qquad \qquad \qquad \qquad \text {mirror boundary},\\ I_P \otimes F_N \mathrm{diag}(\lambda N^2 \mathrm{d}_N^{\mathrm{per}} + \tfrac{1}{\tau }) ^{-1} \bar{F}_N &{} \\ \qquad \qquad \qquad \qquad \qquad \qquad \text {periodic boundary}. \end{array}\right. } \end{aligned}$$

The Schur complement reads as

$$\begin{aligned} S&= \Big (\lambda D_\mathrm{f}^{\scriptscriptstyle {\text {T}}}D_\mathrm{f} + \tfrac{1}{\tau } I \Big ) - \lambda ^2 D_\mathrm{f}^{\scriptscriptstyle {\text {T}}}D_\mathrm{m} X^{-1} D_\mathrm{m}^{\scriptscriptstyle {\text {T}}}D_\mathrm{f}\\&= \Big (\lambda D_P D_P^{\scriptscriptstyle {\text {T}}}\otimes I_N + \tfrac{1}{\tau } I \Big ) \\&\ \ \ \ - \lambda ^2 \big (D_P \otimes D^{\scriptscriptstyle {\text {T}}}\big ) \Big ( I_P \otimes \Big (\lambda N^2 D D^{\scriptscriptstyle {\text {T}}}+\tfrac{1}{\tau } I \Big )^{-1} \Big )\\&\qquad \Big (D_P^{\scriptscriptstyle {\text {T}}}\otimes D \Big )\\&= \Big (\lambda D_P D_P^{\scriptscriptstyle {\text {T}}}\otimes I_N + \tfrac{1}{\tau } I \Big ) \\&\ \ \ \ - \lambda ^2 \Big ( D_P D_P^{\scriptscriptstyle {\text {T}}}\otimes D^{\scriptscriptstyle {\text {T}}}\big (\lambda D D^{\scriptscriptstyle {\text {T}}}+ \tfrac{1}{\tau } I_N \big )^{-1} D \Big )\\&= \lambda D_P D_P^{\scriptscriptstyle {\text {T}}}\otimes \left( I_N - \lambda D^{\scriptscriptstyle {\text {T}}}\big (\lambda D D^{\scriptscriptstyle {\text {T}}}+ \tfrac{1}{\tau } I_N \big )^{-1} D \right) + \frac{1}{\tau } I. \end{aligned}$$

By (21) we have

$$\begin{aligned} \big (D_N^{\mathrm{per}} \big )^{\scriptscriptstyle {\text {T}}}= N F_N \mathrm{diag}\Big (-1 + {\,\mathrm {e}}^{+2\pi \mathrm {i}k/N} \Big )_k \bar{F}_N \end{aligned}$$

and

$$\begin{aligned} D_N ^{\mathrm{per}}= N F_N \mathrm{diag}\Big (-1 + {\,\mathrm {e}}^{-2\pi \mathrm {i}k/N} \Big )_k \bar{F}_N \end{aligned}$$

so that we obtain for periodic boundary boundary conditions

$$\begin{aligned} I_N - \lambda \Big (D_N^{\mathrm{per}}\Big )^{\scriptscriptstyle {\text {T}}}\Big (\lambda D_N^{\mathrm{per}} \Big (D_N^{\mathrm{per}} \Big )^{\scriptscriptstyle {\text {T}}}+ \tfrac{1}{\tau } I_N \Big )^{-1} D_N^{\mathrm{per}} \\ = F_N \mathrm{diag}\left( 1+ \tau \tfrac{1}{\lambda N^2} \mathrm{d}^{\mathrm{per}}_N \right) ^{-1} \bar{F}_N. \end{aligned}$$

Therewith it follows with (24)

$$\begin{aligned} S&= S_{P-1} \mathrm{diag}\Big (\lambda P^2 \mathrm{d}^{\mathrm{zero}}_{P-1} \Big ) S_{P-1} \\&\ \ \ \ \otimes F_N \mathrm{diag}\left( 1+ \tau \tfrac{1}{\lambda N^2} \mathrm{d}^{\mathrm{per}}_N \right) ^{-1} \bar{F}_N + \tfrac{1}{\tau } I\\&= \Big (S_{P-1} \otimes F_N\Big ) \mathrm{diag}\Big (\lambda P^2 \mathrm{d}^{\mathrm{zero}}_{P-1} \\&\ \ \ \ \otimes \Big (1+ \tau \tfrac{1}{\lambda N^2} \mathrm{d}^{\mathrm{per}}_N \Big )^{-1} + \tfrac{1}{\tau } \Big ) \Big (S_{P-1} \otimes \bar{F}_N\Big ) \end{aligned}$$

which yields the assertion for $S^{-1}$ in the periodic case.

For mirror boundary conditions we compute using (23)

$$\begin{aligned} S&= \Big (S_{P-1} \otimes C_N^{\scriptscriptstyle {\text {T}}}\Big ) \mathrm{diag}\Big (\lambda P^2 \mathrm{d}^{\mathrm{zero}}_{P-1} \\&\ \ \ \otimes \Big (1+ \tau \tfrac{1}{\lambda N^2} \mathrm{d}^{\mathrm{mirr}}_N \Big )^{-1} + \tfrac{1}{\tau }\Big ) \Big (S_{P-1} \otimes C_N\Big ) \end{aligned}$$

and inverting this matrix finishes the proof. $\square$

Discretization for three spatial dimensions + time For RGB images of size $N_1 \times N_2 \times N_3$, where $N_3 = 3$, we have to work in three spatial dimensions. Setting $N :=(N_1,N_2,N_3)$, $j :=(j_1,j_2,j_3)$ and defining the quotient $\tfrac{j}{N}$ componentwise we obtain

• $f_i = \bigg (f_i \Big ( \tfrac{j-1/2}{N}\Big ) \bigg )_{j=(1,1,1)}^{N} \in \mathbb R^{N_1,N_2,N_3}$, $i= 0,1$,

• $f = \bigg ( f \Big ( \tfrac{j-1/2}{N}, \tfrac{k}{P} \Big ) \bigg )_{j={(1,1,1)},k=1}^{N,P-1} \in \mathbb R^{N_1,N_2,3,P-1}$,

• $m = (m_1, m_2, m_3)$, with

  $$\begin{aligned}&\bigg (m_1 \Big (\tfrac{j_1}{N_1},\tfrac{j_2-1/2}{N_2},\tfrac{j_3-1/2}{3},\tfrac{k-1/2}{P} \Big )\bigg )_{j_1=1,j_2=1,j_3=1,k=1}^{N_1-1,N_2,3,P}\\&\in \mathbb R^{N_1-1,N_2,3,P},\\&\bigg (m_2\Big (\tfrac{j_1-1/2}{N_1},\tfrac{j_2}{N_2},\tfrac{j_3-1/2}{3},\tfrac{k-1/2}{P}\Big )\bigg )_{j_1=1,j_2=1,j_3=1,k=1}^{N_1,N_2-1,3,P}\\&\in \mathbb R^{N_1,N_2-1,3,P},\\&\bigg (m_3\Big (\tfrac{j_1-1/2}{N_1},\tfrac{j_2-1/2}{N_2},\tfrac{j_3}{3},\tfrac{k-1/2}{P}\Big )\bigg )_{j_1=1,j_2=1,j_3=0,k=1}^{N_1,N_2,2,P}\\&\in \mathbb R^{N_1,N_2,3,P}. \end{aligned}$$

In the definition of m we take the periodic boundary for the third spatial direction into account. Analogously as in the one-dimensional case, when reshaping m and f into long vectors, the interpolation and differentiation operators can be written using tensor products. For the interpolation operator we have

$$\begin{aligned} S_{\text {m}}m&= \begin{pmatrix} (I_P \otimes I_3\otimes I_{N_2} \otimes S_{N_1}^{\scriptscriptstyle {\text {T}}}) m_1\\ (I_P \otimes I_3\otimes S_{N_2}^{\scriptscriptstyle {\text {T}}}\otimes I_{N_1}) m_2\\ (I_P \otimes S_3^{\scriptscriptstyle {\text {T}}}\otimes I_{N_2} \otimes I_{N_1}) m_3\end{pmatrix} \end{aligned}$$

and

$$\begin{aligned} S_{\text {f}}f = \Big (S_P^{\scriptscriptstyle {\text {T}}}\otimes I_3\otimes I_{N_2} \otimes I_{N_1} \Big ) f, \end{aligned}$$

which means, that $S_{\text {m}}m$ computes the average of $m_i$ with respect to the i-th coordinate, $i=1,2,3$, and $S_{\text {f}}f$ computes the average of f with respect to the time variable. Similarly we generalize the difference operator. Then, reordering f and m into large vectors, the matrix form of the operator A is

$$\begin{aligned} A = \Big ( I_P\otimes I_3\otimes I_{N_2}\otimes D_{N_1}^{\scriptscriptstyle {\text {T}}}\, \Big | \, I_P\otimes I_3\otimes D_{N_2}^{\scriptscriptstyle {\text {T}}}\otimes I_{N_1}\, \Big | \,\\ I_P\otimes (D_3^{\mathrm{per}})^{\scriptscriptstyle {\text {T}}}\otimes I_{N_2}\otimes I_{N_1} \, \Big | \, D_P\otimes I_3\otimes I_{N_2}\otimes I_{N_1} \Big ) \end{aligned}$$

so that $AA^{\scriptscriptstyle {\text {T}}}$ reads as

$$\begin{aligned} A A^{\scriptscriptstyle {\text {T}}}&= I_P\otimes I_3\otimes I_{N_2}\otimes D_{N_1}^{\scriptscriptstyle {\text {T}}}D_{N_1}\\&\ \ \ \ +I_P\otimes I_3\otimes D_{N_2}^{\scriptscriptstyle {\text {T}}}D_{N_2}\otimes I_{N_1}\\&\ \ \ \ +I_P\otimes (D_3^{\mathrm{per}})^{\scriptscriptstyle {\text {T}}}(D_3^{\mathrm{per}})\otimes I_{N_2}\otimes I_{N_1}\\&\ \ \ \ +D_P^{\scriptscriptstyle {\text {T}}}D_P\otimes I_3\otimes I_{N_2}\otimes I_{N_1}\\&= \left( C^{\scriptscriptstyle {\text {T}}}_P \otimes \bar{F_3} \otimes C_{N_2-1}^{\scriptscriptstyle {\text {T}}}\otimes C_{N_1-1}^{\scriptscriptstyle {\text {T}}}\right) \mathrm{diag}(\mathrm{d})\\&\ \ \ \ \left( C_P \otimes F_3 \otimes C_{N_2-1}\otimes C_{N_1-1}\right) , \end{aligned}$$

where

$$\begin{aligned} \mathrm{d}&:=I_{3 P (N_2-1)} \otimes N_1^2 \mathrm{diag}\Big (\mathrm{d}_{N_1-1}^{\text {mirr}} \Big )\\&\ \ \ \ + I_{3P}\otimes {N_2}^2 \mathrm{diag}\Big (\mathrm{d}_{{N_2}-1}^{\text {mirr}}\Big )\otimes I_{N_1-1}\\&\ \ \ \ + I_P \otimes 3^2 \mathrm{diag}\Big (\mathrm{d}_{3}^{\text {per}}\Big ) \otimes I_{(N_2-1)(N_1-1)}\\&\ \ \ \ + P^2 \mathrm{diag}\Big (\mathrm{d}_{P}^{\text {mirr}} \Big )\otimes I_{3P(N_2-1)(N_1-1)}. \end{aligned}$$

For the three-dimensional spatial setting we have to solve a four-dimensional Poisson equation, which can be handled separately in each dimension. For the constrained problem, this can be computed using fast cosine and Fourier transforms with a complexity of $\mathcal {O}(N_1 N_2 P\log (N_1 N_2 P))$.