Introduction

The reconstruction of an image from a sparse subset of all pixels is known as inpainting [1]. For the application of image compression, the selection of the data has to be optimised. Let us emphasise that this is by no means a simple task. Selecting 5% of the pixels from a \(256\times {}256\) pixel image offers more than \(10^{5000}\) possible choices. In [27], corresponding strategies are devised via partial differential equations (PDEs). Related models for the inpainting step using the Allen–Cahn model have also been considered in [8], whereas the authors of [9] analysed the Cahn–Hilliard equation. Finally, a broader discussion on fluid dynamics for image reconstruction tasks has been discussed in [10]. The results from [2, 6] motivated the authors of [11] to suggest an optimal control-based approach with a relaxed formulation of the Laplace equation given for known data f by

$$\begin{aligned} c\left( {\mathbf {x}}\right) \left( u\left( {\mathbf {x}}\right) - f\left( {\mathbf {x}}\right) \right) + \left( 1-c\left( {\mathbf {x}}\right) \right) \left( -\Delta \right) u\left( {\mathbf {x}}\right) = 0 \end{aligned}$$
(1)

and additional boundary conditions. The support of the function c indicates that the data locations used for reconstruction should be minimised while preserving a good reconstruction quality. In [11], a local contrast enhancing effect was also observed if c maps to values outside of [0, 1]. Based on [6], this finding was reinforced in [12] where an equivalence with a tuning of the data f was proven. A concrete explanation for this behaviour was not given. However, the understanding of the influence of c on the reconstruction u is crucial for the understanding and improvement of current and future approaches to optimise the inpainting data.

Our Contributions. We show that (1) is related to the Helmholtz equation with a non-constant refraction index if \(c({\mathbf {x}})>1\) and deduce from this interpretation the benefits of non-binary-valued functions c and thus the observations in [11].

In addition, we provide details on how to maximise the benefits gained from this insight. As discussed in [13], it is important to assert that solutions of (1) exist and are unique for each feasible choice of c, and 8 / 7 is stated as its upper bound. To improve this finding, we formulate the finite difference discretisation as an optimisation task. This allows us to specify larger feasible ranges for the values of c than in [5, 13]. We obtain more accurate reconstructions and a stronger contrast enhancing effect.

Inpainting and the Helmholtz equation

Let us briefly recall the mechanisms of inpainting with homogeneous diffusion (IHD). Let \(f:\Omega \rightarrow \mathbb {R}\) be a smooth function on some bounded and open domain \(\Omega \subset \mathbb {R}^{2}\) with a sufficiently regular boundary \(\partial \Omega\). Moreover, let us assume that there exists a closed non-empty set of known data \(\Omega _{K}\subsetneqq \Omega\) that we interpolate. IHD considers the following PDE with mixed boundary conditions:

$$\begin{aligned} \begin{array}{ll} & -{}\Delta u = 0\ {\text {on}}\ \Omega \setminus \Omega _{K},\\ &{} {\text {with}}\quad u = f\ {\text {on }}\ \Omega _{K}\,,\quad {\text {and}}\quad \partial _n u = 0\ {\text {on }}\ \partial \Omega \setminus {}\partial \Omega _{K} \end{array} \end{aligned}$$
(2)

and where \(\partial _{n} u\) denotes the derivative of u in the outer normal direction. We refer to [14] for an extensive study on the existence and uniqueness of solutions.

Following [6], we introduce the confidence function \(c:\Omega \rightarrow \mathbb {R}\) indicating the presence of data. We set \(c({\mathbf {x}})\) to 1 for \({\mathbf {x}}\in \Omega _{K}\) and 0 otherwise. Lets us rewrite (2) as

$$\begin{aligned} c\left( {\mathbf {x}}\right) \left( u\left( {\mathbf {x}}\right) - f\left( {\mathbf {x}}\right) \right) - \left( 1-c\left( {\mathbf {x}}\right) \right) \Delta u\left( {\mathbf {x}}\right) =0\ {\text {on}}\ \Omega \end{aligned}$$
(3)

with Neumann boundary conditions along \(\partial \Omega \setminus {}\partial \Omega _{K}\). As shown in [2, 6], the choice of c has a substantial influence on the solution. Interestingly, (3) also makes sense when c is not binary-valued but takes values in \(\mathbb {R}\). This has been exploited in [11], where (3) is complemented by a convex energy to obtain a sparse and optimal support for c.

Let us now combine the idea of a non-binary-valued confidence function c with the mixed boundary value problem given in (2). We consider:

$$\begin{aligned} \begin{array}{ll} c\left( {\mathbf {x}}\right) \left( u\left( {\mathbf {x}}\right) - f\left( {\mathbf {x}}\right) \right) - \left( 1-c\left( {\mathbf {x}}\right) \right) \Delta u\left( {\mathbf {x}}\right) = 0\ {\text {on }}\ \Omega \setminus {}\Omega _{K},\\ {\text {with}}\quad u = f\ {\text {on }}\ \Omega _{K},\quad {\text {and}}\quad \partial _n u\left( {\mathbf {x}}\right) = 0\ {\text {on }}\ \partial \Omega \setminus {}\partial \Omega _{K}. \end{array} \end{aligned}$$
(4)

Proposition 1

If we define \(\Omega _{K}:=\{x\in \Omega \ |\ c\left( x \right) \equiv 1\}\), then (4) is equivalent to (3) for non-binary-valued c and equivalent to (2) for binary-valued c with range \(\left\{ 0,1\right\}\).

The major difference between (4) and the previous formulations lies in the distinction between \(c = 1\) and \(c \ne 1\). We now proceed to the first important finding of this paper.

Theorem 1

If the confidence function c from ( 3) is continuous, then the inpainting equation from (4) corresponds to the Helmholtz equation in those regions where \(c(x)>1\).

Proof

Due to the intermediate value theorem and the fact that the level sets where \(c\equiv 1\) form closed contours around the regions, where \(c\left( x \right) >1\), we can subdivide \(\Omega \setminus {}\Omega _{K}\) into disjoint regions, where \(c<1\) and where \(c>1\). These regions are separated by \(\Omega _{K}\) on which the solutions u are enforced to coincide with f. Thus, the problem decouples and allows us to discuss these two cases independently. The case \(0\leqslant {}c\left( x \right) <1\) has already been discussed in [13] and will not be investigated further in this paper. Inside those regions, where \(c>1\), we can divide (4) by \(1-c\left( {\mathbf {x}} \right)\) and obtain the following formulation:

$$\begin{aligned} \begin{array}{ll} \Delta u\left( {\mathbf {x}}\right) + \eta ^{2}\left( {\mathbf {x}}\right) u\left( {\mathbf {x}}\right) = g\left( {\mathbf {x}} \right) {\text {on}}\quad \Omega \setminus {}\Omega _{K},\\ {\text {with}}\quad u = f\ {\text {on }}\ \Omega _{K}\quad {\text {and}}\quad \partial _n u\left( {\mathbf {x}}\right) =0 {\text {on }}\ \partial \Omega \setminus {}\partial \Omega _{K}, \end{array} \end{aligned}$$
(5)

where \(\eta ^{2}\left( {\mathbf {x}}\right) = \frac{c\left( {\mathbf {x}}\right) }{c\left( {\mathbf {x}}\right) -1}\) and \(g\left( {\mathbf {x}} \right) = \eta ^{2}\left( {\mathbf {x}}\right) f\left( {\mathbf {x}}\right)\). Equation (5) is the inhomogeneous Helmholtz equation with a refraction \(\eta\) and mixed boundary conditions [15]. \(\square\)

Theorem 1 gives us valuable insight into the properties of our inpainting equation. The Dirichlet data in \(\Omega _{K}\) in (5) can be interpreted in physical terms as a radiation source. The solutions u model the spread of this radiation inside \(\Omega \setminus {}\Omega _{K}\) and the superposition of radiated waves causes the contrast enhancement. Thus, our observations provide a physically motivated explanation for the equivalence shown in [12] and the usage of mask values outside of the range \(\left[ 0,1\right]\). Furthermore, the largest refraction numbers \(\eta\) are obtained for values of c slightly above 1 (\(\eta \rightarrow \infty\) for \(c\searrow {}1\)). If \(c\rightarrow \infty {}\), then \(\eta \rightarrow {}1\), and the sharpening effect vanishes. This explains why values for c significantly larger than 1 are rarely observed in practice.

To get the best possible results for image reconstructions, it is essential to maximise the admissible range of \(\eta\). At the same time, it should be asserted that the discrete setup is well posed. In [13], the author analysed (3) and provided bounds that guarantee that the discretised PDE in (3) has a unique solution. For standard finite difference schemes, these bounds are given by \(0 \leqslant c_{i} \leqslant 8/7\) for all stencil points \(c_{i}\) and \(0 < c_{i}\) for at least one \(c_{i}\).

Improved schemes for the inpainting equation

We now follow the philosophy to optimise key features of our discrete operator for fixed grid parameters and maximise the feasible range for the refraction directly within the design process. For simplicity, we restrict ourselves to \(3\times {}3\) stencils.

To pursue our goals, we need the 2D Taylor expansion. For a sufficiently smooth function \(f:\mathbb {R}^{2}\rightarrow \mathbb {R}\), the Taylor expansion of order n around \({\mathbf {x}}_{0}\) is given by \(f\left( {\mathbf {x}} \right) \approx \sum _{\left| \mathbf {\alpha }\right| \leqslant {}n} \frac{D^{\mathbf {\alpha }}f\left( {\mathbf {x}}_{0}\right) }{\mathbf {\alpha }!} \left( {\mathbf {x}}-{\mathbf {x}}_{0}\right) ^{\mathbf {\alpha }}\), where we employ multi-index notation on \(\mathbf {\alpha }\in \mathbb {N}^{2}\) and where \(D^{\mathbf {\alpha }}f\left( {\mathbf {x}}_{0}\right)\) denotes the partial derivatives of f evaluated at \(\mathbf {x_{0}}\). Given a regular 2D grid with constant step sizes h in each direction, we use the Taylor approximations in all 8 neighbouring pixels. Each Taylor expansion uses \({\mathbf {x}}_{0}:=\left( x_{0},y_{0}\right)\) as centre. This yields the 9 combinations \(f\left( x_{0} + k h, y_{0} + j h \right) \approx \sum _{\left| \mathbf {\alpha }\right| \leqslant {}n} \frac{D^{\mathbf {\alpha }}f\left( x_{0},y_{0}\right) }{\mathbf {\alpha }!} \left( kh, jh\right) ^{\mathbf {\alpha }}\) with k and \(j\in \left\{ -1,0,1\right\}\) each. In a next step, we express the desired derivative \(D^{\mathbf {\alpha }}f\left( x_{0},y_{0}\right)\) as a linear combination of all these positions in our stencil:

$$\begin{aligned}&D^{\mathbf {\alpha }}f\left( x_{0},y_{0}\right) \approx \lambda _{1} f(x_{0}-h,y_{0}-h)\; +\nonumber \\&\qquad \qquad \qquad \lambda _{2} f(x_{0} - h, y_{0}) + \ldots + \lambda _{9} f(x_{0} + h, y_{0} + h ). \end{aligned}$$
(6)

Inserting corresponding Taylor expansions and performing a comparison of coefficients leads to a linear system of equations. Its solutions represent the coefficient vectors \(\mathbf {\lambda }\) for the stencil. Omitting, for clarity, the common argument \((x_{0}, y_{0})\), we have

$$\begin{aligned}&D^{\mathbf {\alpha }}f = \sum _{j=1}^{9} \lambda _{j} f\ + \left( \sum _{i=1}^{3}\lambda _{6+i} - \lambda _{i} \right) h f_{x}\ + \nonumber \\&\qquad \qquad \left( \sum _{i=0}^{2} \lambda _{3(i+1)} - \lambda _{1+3i} \right) h f_{y} + \left( \sum _{i=1}^{3}\lambda _{i} +\lambda _{6+i} \right) \frac{h^{2}}{2} f_{xx}\ + \nonumber \\&\qquad \qquad \qquad \left( \sum _{i=0}^{1}\lambda _{7+2i}-\lambda _{1+2i} \right) h^{2} f_{xy} + \left( \sum _{i=0}^{2} \lambda _{1+3i} + \lambda _{3(i+1)} \right) \frac{h^{2}}{2} f_{yy} \end{aligned}$$
(7)

up to \(\mathcal {O}(h^{3})\). Unfortunately, this approach does not lead, in general, to a square system matrix. The number of unknowns coincides with the number of stencil positions. On the other hand, if we perform a Taylor expansion up to order n, we obtain \(n(n+1)/2\) equations. These numbers rarely match. Nevertheless, unless the equations contradict, it is still possible to determine a particular solution \(\mathbf {\lambda } = \left( \lambda _{i}\right) _{i=1}^{9}\) as well as a basis \(\left\{ \mathbf {\nu }_{1}, \mathbf {\nu }_{2}, \ldots \right\}\) for the nullspace of the matrix and express all solutions as \(\mathbf {\lambda } + \sum _{j} \beta _{j} \mathbf {\nu }_{j}\).

By computing a particular solution of the linear system for an approximation of the second-order derivatives \(f_{xx}\), we arrive at the parametric representations presented in Fig. 1. The stencil for \(f_{yy}\) is obtained analogously and corresponds to transposing the stencil of \(\partial _{xx}f\). The stencil for the Laplacian as in Fig. 1 is obtained by adding the stencils for \(f_{xx}\) and \(f_{yy}.\)

Fig. 1
figure 1

Differential operators and corresponding stencils. The free parameter \(\beta\) stems from the fact that the nullspace of the matrix is one dimension

Let us now compute the stencil entries for our inpainting task from (3) resp. (4). As discussed in [11], these equations can be discretised and written as follows:

$$\begin{aligned} \mathbf {A}\left( \mathbf {c}\right) \mathbf {u} := \left( {{\mathrm{diag\!}}}\left( \mathbf {c} \right) + \left( \mathbf {I} - {{\mathrm{diag\!}}}\left( \mathbf {c} \right) \right) \left( -\mathbf {L} \right) \right) \mathbf {u} = {{\mathrm{diag\!}}}\left( \mathbf {c}\right) \mathbf {f} \end{aligned}$$
(8)

where \(\mathbf {c}\), \(\mathbf {u}\), and \(\mathbf {f}\) are the discretised variants of c, u, and f respectively. The matrix \(\mathbf {L}\) is the discrete analogue of the Laplace operator with incorporated boundary conditions, whereas \(\mathbf {I}\) is the identity matrix. We assume that \(\mathbf {L}\) is discretised with the stencil from Fig. 1. The system matrix of (8) is large and banded. A straightforward computation for non-boundary pixels i shows that the stencil is given as in Fig. 2.

Fig. 2
figure 2

Stencil for the inpainting matrix \(\mathbf {A}\left( \mathbf {c}\right)\) from (8) for a non-boundary pixel at position i. These stencil entries correspond to the non-zero entries of \(\mathbf {A}\left( \mathbf {c}\right)\) in the i-th row. Along the image boundaries, the stencil has to be adapted to consider the boundary conditions

For \(\beta =-1/2\), the corresponding stencils for the Laplacian as well as for the inpainting matrix perform an undesirable odd-even decoupling. As a remedy, values for \(\beta\) should be chosen in the range \(\left[ -1,-1/2\right)\). If \(\beta =-1\), we obtain the well-known standard five point stencil for the Laplacian.

Following [5, 13], we can use the stencil entries to obtain estimates on the \(c_{i}\) for which invertibility of the system matrix is asserted below. This finding extends results in [5] with feasible range \(\left\{ 0,1\right\}\) and from [13] with feasible range \(\left[ 0,8/7\right]\).

Theorem 2

The inpainting matrix \(\mathbf {A}\left( \mathbf {c} \right)\) from (8) corresponding to the stencil from Fig.  2 is invertible when all \(c_{i}\) lie in the range \(\left[ 0,4/3\right]\) for \(h=1\) and \(\beta =-1/2\). This is the largest possible range for the Laplacian from Fig. 1. Furthermore, the lower bound \(c_{i} \geqslant 0\) for the mask entries \(c_{i}\) can only be asserted when \(\beta \in \left[ -1,-1/2\right]\).

Proof

We follow the ideas from [5, 13]. By applying Geršgorin’s circle theorem at the rows of the matrix \(\mathbf {A}\left( \mathbf {c} \right)\), we obtain pointwise estimates for the position of its eigenvalues. We note that all non-zero entries in any of the rows of \(\mathbf {A}\left( \mathbf {c} \right)\) are given by the stencil entries from Fig. 2. To exclude 0 from the spectrum, we must solve

$$\begin{aligned} c_{i} + 4\left( c_{i}-1\right) - 4 \left( \left| \left( -2 \beta - h^{-2} \right) \left( c_{i}-1 \right) \right| + \left| \left( \beta + h^{-2} \right) \left( c_{i}-1 \right) \right| \right) > 0. \end{aligned}$$
(9)

For \(h=1\), a lengthy but simple computation shows that \(0< c_{i} < 8\beta /(1+8\beta )\) must hold whenever \(\beta \leqslant -1/2\). Other values for \(\beta\) yield ranges which do not include the interval (0, 1] and thus are of no interest to us. Maximising the upper bound is possible for \(\beta = -1/2\) and yields the range (0, 4 / 3]. An identical analysis for the cases where a pixel is placed along a boundary or in a corner does not yield further restrictions. \(\square\)

Let us now demonstrate the benefits of larger feasible ranges for the mask values c. Our objective in this is to improve image reconstructions by IHD. Thus, we consider for quantitative evaluation a small synthetic image and measure the reconstruction error in function of the mask values. Our test image consists of \(25\times {}25\) pixels representing a sampled version of the function \(x^{2}+y^{2}\) over \(\left[ 0,1\right] \times \left[ 0,1 \right]\). Our mask contains 5 non-zero entries. In each corner, we fix the mask value to be 1. The remaining non-zero entry is placed at the centre of the image and we measure the mean-squared error (MSE) of the reconstruction as a function of the mask value at this position. We study this setup for different values \(\beta\), as shown in Fig. 3.

Fig. 3
figure 3

MSE for different stencil choices in function of the mask value. For each stencil, the MSE is monotonically decreasing. Most stencils yield similar errors. The huge gap in the error between the stencil for \(\beta =-1/2\) and all the others is due to the odd-even decoupling of the corresponding stencil. The vertical lines indicate the maximal allowed value of c for each case

In a second experiment, we show that our discretisations allow a contrast enhancing effect. To this end, we select a small \(10\times {}12\) grey-scale image patch with pixel value 0.25 on the left half and 0.75 on the right half. Our masks consists of a small strip with 2 pixels width along the image edge. We set all non-zero mask values to the same value \(8\beta /(1+8\beta )\), and let \(\beta\) vary in the admissible range \(\left[ -1,-1/2\right]\). For each value of \(\beta\), we consider the reconstruction at the two neighbouring pixels from each side of the image. Table 1 confirms the desired contrast improvements.

Table 1 Possible contrast enhancement for different stencils. Values \(\beta\) in \(\left( -1,-1/2\right]\) correspond to our discretisations with enhanced contrast properties. The improved contrast is clearly visible for increasing \(\beta\)

Conclusions

This paper shows the relation between the classic Helmholtz equation and the inpainting problem. Thus, we relate two up to now unconnected fields from science. In the future, we hope to develop better performing inpainting models based on this insight.