An Edge-Preserving Regularization Model for the Demosaicing of Noisy Color Images

Abstract

This paper proposes an edge-preserving regularization technique to solve the color image demosaicing problem in the realistic case of noisy data. We enforce intra-channel local smoothness of the intensity (low-frequency components) and inter-channel local similarity of the depth of object borders and textures (high-frequency components). Discontinuities of both the low-frequency and high-frequency components are accounted for implicitly, i.e., through suitable functions of the proper derivatives. For the treatment of even the finest image details, derivatives of first, second, and third orders are considered. The solution to the demosaicing problem is defined as the minimizer of an energy function, accounting for all these constraints plus a data fidelity term. This non-convex energy is minimized via an iterative deterministic algorithm, applied to a family of approximating functions, each implicitly referring to geometrically consistent image edges. Our method is general because it does not refer to any specific color filter array. However, to allow quantitative comparisons with other published results, we tested it in the case of the Bayer CFA and on the Kodak 24-image dataset, the McMaster (IMAX) 18-image dataset, the Microsoft Demosaicing Canon 57-image dataset, and the Microsoft Demosaicing Panasonic 500-image dataset. The comparisons with some of the most recent demosaicing algorithms show the good performance of our method in both the noiseless and noisy cases.

1 Introduction

The demosaicing problem arises from acquiring RGB color images through CCD (charged coupled devices) digital cameras. In the RGB model, each pixel of a digital color image is associated with a triplet of numbers, representing the intensity of the red, green, and blue, respectively. However, most commercial cameras employ a single sensor associated with a color filter that only permits, at each pixel, the measurement of the reflectance of the scene at one of the three colors, according to a predefined scheme or pattern, called color filter array (CFA), as illustrated in Fig. 1. This situation implies that, for each pixel, the other two missing colors must be estimated. Although several diverse CFAs have been proposed for the acquisition (e.g., see [31] and [35] for the square pixel layout and [3] for the Penrose aperiodic pixel layout), the most largely diffuse is the Bayer pattern [4], shown in Fig. 2. Most of the literature on demosaicing is thus devoted to algorithms designed with explicit reference to the Bayer pattern. In [30, 46], and [56], comprehensive surveys of the state of the art can be found. In the Bayer scheme, green is sampled in a number of pixels double the number where red and blue are measured to exploit the human eye’s higher sensibility to the green wavelength.

Fig. 1

Color image acquisition based on a CFA

Fig. 2

The Bayer pattern

By decomposing the recorded digital image into the three color channels, one obtains three downsampled grayscale images so that demosaicing could be interpreted as the problem of interpolating grayscale images from sparse data.

In most recreational cameras, demosaicing is part of the processing pipeline required to convert these images into a viewable format. The camera’s built-in firmware is substantially based on very fast, or even real-time, interpolation algorithms.

Nevertheless, interpolation is not sufficient to produce the high-quality images required, e.g., for professional purposes. Indeed, interpolation is often applied to each channel separately, disregarding the mutual solid correlation between the channels. Since interpolation algorithms are basically low-pass filters, they perform well in the low-frequency components but reduce the high-frequency ones, thus producing color artifacts in the output image. A remedy to this problem should consider the high correlation among the three color channels, which occurs mainly in correspondence with their high-frequency components. Indeed, the image edges, such as object borders and textures, are largely shared by the color channels. Unavoidably, exploiting the inter-channel information requires more complex algorithms, which are usually time-consuming and cannot run onboard.

Nowadays, many professional digital cameras can save images in a raw format, thus allowing users to demosaic them using offline software. When the raw image data is accessible, one can use various demosaicing algorithms instead of being limited to the one built into the camera. For example, the raw development program RawTherapee [64] gives the user an option to choose, among around ten different algorithms, which should be used. For the Bayer CFA, AMaZE (Aliasing Minimization and Zipper Elimination), developed by Emil J. Martinec in 2010, yields the best results in most cases. Nevertheless, since different demosaicing algorithms present differences in rendering the finest detail and grain texture, photographers often prefer a particular algorithm for aesthetic reasons related to this effect. On the other hand, when it comes to professional applications, computational time is less critical. Thus, there is an interest in developing dedicated and even very sophisticated algorithms that can provide demosaiced images where the high-frequency components are accurately reconstructed.

To this end, this paper proposes an algorithm for image demosaicing that falls within the framework of the edge-preserving regularization approaches and is suited, naturally, to deal with noisy data. More precisely, we propose an algorithm for joint demosaicing and denoising. Regularization requires the adoption of constraints for the solution. The constraints we consider here are intra-channel (spatial) and inter-channel (spectral) local correlation. Concerning the intra-channel correlation, we assume the intensity of each channel to be locally regular, i.e., piecewise smooth, so that noise can also be removed. We describe this constraint through stabilizers that discourage intensity discontinuities of first, second, and third order in a selective way, i.e., permitting to discontinuities associated with actual edges to emerge. This allows us to describe scenes that are even very complex. Indeed, first-order local smoothness characterizes images consisting of constant patches, second-order local smoothness describes patches whose pixels have values varying linearly, while third-order local smoothness is used to represent images made up of quadratic-valued patches. As per the inter-channel correlation, we enforce it in correspondence with the intensity discontinuities using constraints that promote their amplitude in the three channels to be equal almost everywhere.

Note that all these constraints are not biased in favor of one of the three channels, nor is the geometry of the sampling pattern in any way exploited. Thus, the method we propose is entirely independent of the CFA considered, although, in the experimental result section, we present its application to images mosaiced through the Bayer CFA.

All the above constraints, including the data fidelity term, are merged in a non-convex energy function, whose minimizer is our desired solution.

The non-convexity characteristics of the energy function preclude its minimization through conventional gradient descent optimization techniques. Stochastic and deterministic methodologies offer alternative avenues for minimizing non-convex energy functions. While stochastic techniques often yield exact outcomes, their computational demands are considerable (cf. [25]). Conversely, deterministic algorithms, while not guaranteeing convergence to the global optimum, facilitate the attainment of satisfactory reconstructions within reduced computational durations (cf. [6, 8]).

Here, we adopt the graduated non-convexity (GNC) algorithm [9], which is a prevalent one within the deterministic approaches. This technique approximates the energy function through progressively refined approximation functions converging toward the original function. Each approximation function is optimized using conventional optimization algorithms, leveraging the minima attained in the preceding approximation as initial values.

As mentioned, our edge-preserving regularization approach, combined with the GNC minimization strategy, can produce image solutions that exhibit reliable and geometrically consistent discontinuities of both the intensity and the gradients despite the necessary smoothness constraints.

In the first works proposing edge-preserving regularization, the image discontinuities were often represented by extra, explicit variables, the so-called line processes [25]. That way, it was relatively easy to formulate their required properties regarding constraints. Nevertheless, the use of explicit line variables entails significant computational costs. Thus, so-called duality theorems were derived (see, e.g., [15, 26, 14]) to demonstrate the edge-preserving properties of suitable stabilizers without introducing extra variables. In particular, we developed duality theorems to determine the properties required for stabilizers to manage lines with the desired regularity features implicitly. In this paper, we choose a suitable family of approximations for the GNC having the peculiarity that each function satisfies the conditions required for an implicit treatment of geometrically significant edges, as expressed in the duality theorems proposed in [14]. This allows for better adherence of the approximations to the ideal energy function, resulting in better coherence with the properties required for the desired solution.

GNC, initially introduced in [9], has seen significant advancements over the years. In [7], the GNC framework was extended to accommodate the restoration of noisy images, by incorporating considerations on the geometry of discontinuities. In [60], blur in noisy images has been considered as well. Since then, [15] proposed GNC for deblurring and denoising tasks, enforcing constraints such as line continuation or line non-parallelism, and a GNC variant was designed to deal with point spread functions with a vast domain [12].

The GNC methodology has found applications across diverse domains, which stand as a testament to its versatility and efficacy. Indeed, it has been successfully utilized in addressing the combinatorial data analysis challenge of seriation, as evidenced in [23]. Moreover, its utility extends to tackling stochastic problems, as discussed in [32], and resolving combinatorial optimization dilemmas defined on the domain of partial permutation matrices, as explored in [48]. Additionally, the GNC framework has been harnessed for addressing the maximum a posteriori inference problem, elucidated in [49], as well as for tasks such as pose estimation, documented in [67], and spatial perception, outlined in [71].

This paper extends the GNC methodology by explicitly focusing on the non-parallelism constraint for the discontinuities and proposing a first, initial approximation that is componentwise convex. We call this new version of GNC graduated componentwise non-convexity (GCNC).

The paper is organized as follows. In Sect. 2, state-of-the-art data in the color image demosaicing and denoising field are surveyed. Section 3 is devoted to formulating the problem and adopting the specific edge-preserving regularization strategy. In Sect. 4, the solution algorithm is described in detail. Section 5 is devoted to the quantitative comparison between the results obtained with our method and those of some of the most performing algorithms proposed in the recent literature, using the Kodak image dataset [43], the McMaster image dataset [75], the Microsoft Demosaicing Canon Dataset [38], and the Microsoft Demosaicing Panasonic Dataset [38] as benchmark sets, and with specific reference to the Bayer CFA. Conclusions and suggestions for prospects are given in Sect. 6, and finally, in the Appendices, some mathematical aspects are developed in detail.

2 Related Work

A significant problem of demosaicing is avoiding over-smoothing of the edges, so the fundamental feature of any method is its ability to perform interpolation along and not across the edges. Some methods perform directional interpolation by analyzing the variance of the color differences to exploit the high correlation between the color planes [18]. For example, the work in [2] proposes high-order interpolation and Sobel operators to compute the gradients, and in [24], a level set method is used to minimize an energy function that gives the direction of the edges. In [19], the interpolation direction is chosen by exploiting an edge-sensing parameter called integrated gradient, which simultaneously considers color intensity and color difference. In general, due to solid correlations existing among three color channels in nature, performing interpolation in the relatively smooth color difference field can be easier and more convenient.

In other methods, the best reconstruction of the missing data, first estimated by interpolating along horizontal and vertical directions, is chosen [33, 52], or the two reconstructions are fused [74, 75]. In particular, [33] proposes an algorithm based on the Laplacian filter by selecting the interpolation directions having the most minor level of color artifacts. Instead, the method in [17] infers the missing colors by considering the local image geometry through the image self-similarity.

As an alternative to color difference interpolation, the algorithms in [41] and [72], among others, are based on interpolation in a residual domain, where the residuals are differences between observed and tentatively estimated pixel values. This approach is justified because the residual field is smoother than the color difference data field. Hence, the methods based on residual interpolation may be advantageous regarding both peak signal-to-noise (PSNR) and subjective visual quality. In [59], the adaptive residual interpolation method is proposed to combine the residual interpolation [39] and the minimized-Laplacian residual interpolation [40]. The developed method is extensively tested on large datasets in [36].

In [29] and [45], the strong correlation between the high frequencies of the three color components is directly exploited. In particular, the algorithm presented in [29] forces similarity between the high frequencies of the red and the green and of the blue and the green.

The sparse nature of color images has also been exploited for demosaicing. A suitable dictionary is designed and applied with the iterative K-SVD algorithm in [50], whereas in [58], the dictionary is constructed based on a clear distinction between the inter-channel and intra-channel correlations of natural images and the sparse representation of the image is found through compressed sensing. In [1], a locally adaptive approach is used for demosaicing dual-tree complex wavelet coefficients.

Within regularization approaches, the total-variation principle is used in [66], while [54] proposes first a general quadratic smoothness regularizer and then an adaptive filter in order to improve the reconstruction near the edges of the first estimate.

The methods surveyed above have been mainly designed for noise-free data. An abundance of literature has also been devoted to the more realistic noisy data scenario. In this context, performing denoising as a pre-or post-processing has significant drawbacks. In the first case, denoising must be separately performed on the individual channel so that the entire image resolution cannot be exploited. In addition, denoising alters the color samples in the raw image. On the other hand, if demosaicing is performed first, the interpolation process changes the noise distribution. For instance, in [55] [54], the authors evaluate the statistical characteristics of the noise resulting from the demosaicing process performed through space-varying filters, and then design an ad hoc post-processing denoising strategy.

The method in [76] works in two steps. First, the full-resolution green component is recovered from the difference signals of the color channels by exploiting both spectral and spatial correlations to suppress sensor noise and interpolation error simultaneously. Second, the CFA channel-dependent noise is removed from the reconstructed green channel with a wavelet-based approach, and the red and blue channels are also estimated and denoised. The work in [34] presents an algorithm that uses a modified total least squared estimation technique to estimate an ideal demosaicing filter to deal with the noise affecting the data.

In [20], a noisy mosaiced image’s luminance and chrominance channels are first reconstructed by exploiting a frequency analysis of the sampling pattern induced by the Bayer CFA. Wiener filters are then designed to denoise the chrominances, whereas the luminance is linearly filtered as a grayscale image. An extended variant of this approach is also proposed, in which the demosaiced image is mosaiced again and then demosaiced using the method in [74].

Regularization is an ideal setting for joint demosaicing and denoising, which can be performed simultaneously based on the same image priors. These can be manually designed, as in [21], where a total variation (TV) prior ensures the smooth property of the image or can be learned from the image dataset, as in [38], where regression tree fields are used to learn realistic image datasets. The method in [42] models the problem as a minimization problem and uses an image dataset to improve performance.

In recent years, many demosaicing algorithms have been developed by exploiting the potential of convolutional neural networks (CNNs) to avoid dependence on empirically defined priors. For instance, in [27], a deep demosaicing and denoising network was trained on millions of artificially degraded images. A two-stage CNN and a three-stage CNN have been proposed in [69] and [22], respectively, and in [68], an image demosaiced using traditional interpolation schemes is refined using the residual network. The work in [44] a specific network architecture is designed, inspired by powerful classic image regularization and large-scale optimization. To improve the quality of the recovered image, in [47], the guidance of the density map and the exploitation of edge characteristics are proposed, and in [70], the structure of the CNN model and the loss functions are carefully studied. Finally, in [63], a CNN model is used as a prior within a regularization setup.

Although deep learning-based methods have demonstrated higher performance, they are highly data-dependent and require many training images, i.e., full-resolution RGB images. In many approaches, the authors have used as reference images already processed; that is, they apply a mosaic mask on images already demosaiced, thus obtaining unrealistic training pairs [68]. In fact, with this learning strategy, the main problem is that demosaicing the training data artifacts will hamper the reconstruction’s performance and overall quality. Furthermore, methods based on learning require high memory and computation costs, which is undesirable for integrated sensor systems. For these reasons, interpolation-based methods are still widely applied in practical use due to their simplicity.

3 Formulation of the Demosaicing Problem and Its Regularization

3.1 The Data Generation Model

A color image of size $n\times m$ can be represented as a vector $\textbf{x}\in {\mathbb {R}}^{3nm}$, $\textbf{x}=\left( (\textbf{x}^{(r)})^T\, (\textbf{x}^{(g)})^T\,(\textbf{x}^{(b)})^T \right) ^T$, where $\textbf{x}^{(r)}, \textbf{x}^{(g)}, \textbf{x}^{(b)} \in {\mathbb {R}}^{nm}$ are the red, green and blue channels expressed in the lexicographic notation, respectively. The mosaicing problem is formulated as

$$\begin{aligned} \textbf{y} =M \;(\textbf{x}+\textbf{n}), \end{aligned}$$

(1)

where $\textbf{x}, \textbf{y}, \textbf{n} \in [0,255]^{3nm}$ denote the ideal color image, the mosaiced image, and the additive noise, respectively. We assume the noise to be independent, Gaussian, with null mean and variance $\sigma ^2$. The matrix $M\in \{0,1\}^{3nm\times 3nm }$ is a linear operator associated with the acquisition pattern, consisting of the following block diagonal matrix:

$$\begin{aligned} M= \left( \begin{array}{ccc} M^{(r)} &{} O &{} O\\ O &{} M^{(g)}&{} O\\ O &{} O &{} M^{(b)} \end{array} \right) , \end{aligned}$$

(2)

where $O \in {\mathbb {R}}^{nm\times nm}$ is the null matrix, and $M^{(r)}$, $M^{(g)}$, $M^{(b)}$ are diagonal matrices in ${\mathbb {R}}^{nm\times nm}$. For the Bayer pattern, the diagonal elements of these matrices are given by

$$\begin{aligned} \begin{aligned} m_{(i,j),(i,j)}^{(r)}= & {} \left\{ \begin{array}{ll} 1, &{} i \equiv _2 j \equiv _2 0,\\ 0, &{} \text{ otherwise, } \end{array} \right. \\ \hspace{-1mm} m_{(i,j),(i,j)}^{(g)}= & {} \left\{ \begin{array}{ll} 1, &{} i \equiv _2\hspace{-5mm}/ \; \; \;j, \\ 0, &{} \text{ otherwise, } \end{array} \right. \\ m_{(i,j),(i,j)}^{(b)}= & {} \left\{ \begin{array}{ll} 1, &{} i \equiv _2 j \equiv _2 1, \\ 0, &{} \text{ otherwise, } \end{array} \right. \end{aligned} \end{aligned}$$

(3)

where (i, j) is the generic pixel index.

The demosaicing problem is the inverse problem associated with the direct problem formulated in (1) and consists of finding an estimate $\widetilde{\textbf{x}}$ of the ideal image, given the mosaiced image $\textbf{y}$ and the operator M. Since M is singular, the demosaicing problem is ill-posed in the Hadamard sense because, in general, it does not admit a unique solution. Given $\textbf{y}$, there are infinitely many feasible solutions since, at each pixel, the values of the two unmeasured channels do not contribute to the data. Therefore, regularization techniques are necessary to reduce the number of solutions.

3.2 The Regularization Model

We define our regularized solution $\widetilde{\textbf{x}}$ as an argument of the minimum of the following energy function:

$$\begin{aligned} E(\textbf{x})= & {} \left\| M(\textbf{x}-\textbf{y})\right\| _2^2 +\sum _{k=1}^3\sum _{c\in C_k} \varphi \left( N^k_c\textbf{x}, N^k_{p_k(c)}{} \textbf{x} \right) \nonumber \\{} & {} + \sum _{k=1}^3\sum _{c\in C_k} \varphi \left( V^k_c\textbf{x}, V^k_{p_k(c)}{} \textbf{x} \right) , \end{aligned}$$

(4)

where $\left\| \cdot \right\| _2$ denotes the Euclidean norm, and the first term of the right hand of (4) expresses a data fidelity constraint, which is identically null in the noiseless case. The second term in the right hand of (4) regulates the intra-channel smoothness of the involved image. The third term imposes a correlation between the different channels, i.e., an inter-channel smoothness. Intra-channel and inter-channel smoothness are measured through the operators $N_c^k$ and $V_c^k$, respectively, and $\varphi$ is a stabilizer that weights the degree of smoothness required and relaxes it when a discontinuity is expected.

Let us start by analyzing the form of the operator $N_c^k$, given by

$$\begin{aligned} N^k_{c}{} \textbf{x}= \left\| \left( D^k_{c} \textbf{x}^{(r)}, D^k_{c} \textbf{x}^{(g)}, D^k_{c}{} \textbf{x}^{(b)} \right) \right\| _2, \end{aligned}$$

(5)

where $D^k_c$ is a finite difference operator of order k applied to a suitable set c of adjacent pixels, called clique of order k. Therefore, from (5), it appears that $N_c^k$ is the norm of the vector of the finite differences of the intensities of the red, green, and blue channels computed on the clique c of order k. The set $C_k$ collects all cliques of order k. Each of such cliques is uniquely associated with a discontinuity of order k, labeled by a hidden line element.

To reconstruct the finest details in the images, we consider finite differences and then discontinuities of the first, second, and third order, that is $k=1,2,3$. The geometry of the associated cliques is described in Appendix A.

The edges of the first order separate homogeneous patches in the image, the edges of the second order mark the slope of linearly varying areas, and the edges of the third order are associated with the intensity discontinuities in regions where intensity varies quadratically.

As the inter-channel correlation aims to maintain the clue of the objects in the image, the finite difference operators should behave similarly in all three channels. So we define the operator $V_c^k$ as follows:

$$\begin{aligned} V^k_c\textbf{x}= & {} \left\| \left( D^k_{c} \textbf{x}^{(r)} - D^k_{c} \textbf{x}^{(g)}, D^k_{c} \textbf{x}^{(r)} - D^k_{c} \textbf{x}^{(b)},\right. \right. \nonumber \\{} & {} \left. \left. D^k_{c} \textbf{x}^{(g)} - D^k_{c} \textbf{x}^{(b)} \right) \right\| _2, \end{aligned}$$

(6)

which is the norm of the vector of the inter-channel differences of the intra-channel k-order derivatives. Again, a hidden line variable is implicitly associated with the clique c for each order $k=1,2,3$. These further sets of hidden line variables mark the discontinuities between areas having homogeneous clues.

In (4), $N_c^k$ and $V_c^k$ are weighted by suitable stabilizers. These stabilizers should regulate the degree of smoothness required in the two cases and relax it when discontinuities are expected and dependent on their amplitude. In (4), we adopted the same parametric stabilizer $\varphi$ for both the operators $N_c^k$ and $V_c^k$ and let its parameters possibly vary in the two terms (see also [14]).

For a more accurate reconstruction, edges must not be thick, or the object contours must not be blurred equivalently. To this aim, it is advisable to inhibit the creation of discontinuities at two adjacent cliques. Specifically, to prevent double edges of order k, simultaneous discontinuities at the cliques c and the previous one $p_k(c)$ should be inhibited (see Appendix A for the definition of adjacent cliques).

When $p_k(c)$ is not defined for the mixed cliques and the cliques on the border of the image, we automatically assume that the adjacent discontinuity is null.

Having in mind the above-described properties to be featured by the stabilizer, we adopt herein a bivariate function $\varphi : {\mathbb {R}} \times {\mathbb {R}} \rightarrow {\mathbb {R}}$ (see also [15]), defined by

$$\begin{aligned} \varphi (t_1,t_2) =\left\{ \begin{array}{ll} g_1(t_1), \qquad \qquad \qquad \qquad \qquad \qquad \qquad \text { if } |t_2 |\le s,\\ \\ \Bigl ( 1- \frac{2(|t_2 |-s)^2}{(\zeta -s)^2} \Bigl ) \, g_1(t_1) + \frac{2(|t_2 |-s)^2}{(\zeta -s)^2} \, g_2(t_1),\\ \qquad \qquad \qquad \qquad \qquad \qquad \qquad \text { if } s< |t_2 |\le \frac{\zeta +s}{2},\\ \\ \frac{2(|t_2 |-\zeta )^2}{(\zeta -s)^2} \, g_1(t_1) + \Bigl ( 1- \frac{2(|t_2 |-\zeta )^2}{(\zeta -s)^2} \Bigl ) \, g_2(t_1),\\ \qquad \qquad \qquad \qquad \qquad \qquad \qquad \text { if } \frac{\zeta +s}{2}< |t_2 |<\zeta ,\\ \\ g_2(t_1), \qquad \qquad \qquad \qquad \qquad \qquad \qquad \text { if } |t_2 |\ge \zeta , \end{array}\right. \end{aligned}$$

(7)

where

$$\begin{aligned} s=\dfrac{\sqrt{\alpha }}{\lambda }, \end{aligned}$$

(8)

and $\zeta$ is chosen in such a way that $\zeta - s$ is a positive and sufficiently small quantity, and for $i=1, 2$ it is

$$\begin{aligned} g_i(t_1)= & {} \left\{ \begin{array}{ll} \lambda ^{2}t_1^{2}, &{} \text { if } |t_1 |<q_i,\\ \\ \alpha _i -\dfrac{\tau }{2}(|t_1 |-r_i)^{2}, &{} \text { if } \, q_i\le |t_1 |\le r_i, \\ \\ \alpha _i, &{} \text { if } \,|t_1 |>r_i, \end{array} \right. \end{aligned}$$

(9)

$$\begin{aligned} \alpha _i= & {} \left\{ \begin{array}{ll} \alpha , &{} \text {if } i=1, \\ \\ \alpha +\varepsilon , &{} \text {if } i=2, \end{array} \right. \end{aligned}$$

(10)

$$\begin{aligned} q_i= & {} \dfrac{\sqrt{\alpha _i}}{\lambda ^{2}} \left( \dfrac{2}{\tau } + \dfrac{1}{\lambda ^2} \right) ^{-1/2}, \end{aligned}$$

(11)

$\tau$ is a large enough real constant, and

$$\begin{aligned} r_i=\dfrac{\alpha _i}{\lambda ^2 \, q_i}, \quad i=1,2. \end{aligned}$$

(12)

The graph of the function in (7) for specific values of the parameters is shown in Fig. 3 (a). Note that when $\zeta - s=0$ and $\tau$ tends to $+ \infty$, we have an ideal model with Boolean line variables (see Fig. 3 (b)). We have set $\zeta =s+10^{-4}$ and $\tau =10^4$ in our experimental results.

In (7)–(12) $\lambda ^2$ is a regularization parameter, which has the role of determining the smoothness of the solution, $\alpha$ is a cost, which we have to pay whenever we introduce the image discontinuities. When there are some parallel discontinuities, we add the quantity $\varepsilon$ to this cost. Thus, the value s in (8) is just the threshold necessary to introduce a discontinuity in the image when there are no parallel discontinuities.

Fig. 3

(a) $\varphi$ with $\lambda =1$, $\alpha =80$, and $\varepsilon =80$; (b) $\varphi$ in the case of $\zeta =s$ and $\tau =+ \infty$

In general, the stabilizer’s analytical form determines the amplitude of the discontinuities in the reconstructed image by promoting on-off discontinuities of large amplitude above a given threshold or more slowly varying discontinuities of graded amplitudes. In the primal-dual formalism, the first type’s stabilizers implicitly address “hard”, Boolean line elements, while the second type addresses hidden “soft” line elements, ideally valued in [0, 1].

We recall that the functions $g_i$, $i=1,2$, defined in (9), are approximations of class $C^1$ of the classical truncated parabola defined in [9] (see also [10, 14, 15]) that, when used as a stabilizer, implicitly addresses a Boolean line process. In the bivariate case, the function $\varphi$ defined in (7) possesses the same characteristic when $\tau$ tends to $+ \infty$ and $\zeta$ is very close to s. The actual form we propose is an approximation of such a function, with the property of class $C^1$, which is essential for converging the minimization algorithm (see Subsection 4.1).

4 Graduated Componentwise Non-Convexity Algorithm

We observe that the function $\varphi$ defined in (7), as a function of two variables, is not convex, and hence neither is the energy function $E(\textbf{x})$, defined in (4) as a function of $3 \, n \, m$ variables. Thus, to minimize E, we determine a finite family of approximating functions $\{E^{(p)}\}_p$, where $E^{(0)}$ is componentwise convex, and $E^{(2)}$ is the original energy function E. The initial point to minimize the componentwise convex approximation is found using the Local Edge Preserving (LEP) algorithm used in [13] for demosaicing. The LEP is a high-speed algorithm consisting of two phases. In the first phase, the missing components are determined by a weighted mean, which guarantees the preservation of the edges. In the second phase, the differences between the colors of the channels are imposed to be constant within homogeneous areas. Our algorithm in its whole is called graduated componentwise non-convexity (GCNC) and is presented in Algorithm 1.

Algorithm 1

GCNC algorithm

Note that our algorithm can be seen as a variant of the graduated non-convexity (GNC) algorithm (see also [7, 15, 57, 60,61,62, 65]).

To construct the first componentwise convex approximation $E^{(0)}$, we find a componentwise convex approximation for the stabilizers in (7), since the data term in (4) is globally convex. Such componentwise convex approximations can be constructed based on a componentwise convex approximation of the bivariate function $\varphi$, as shown in Appendix D. To do this, we proceed as follows. First of all, we approximate the functions $g_i(t_1)$ with the following convex approximations given by

$$\begin{aligned} \overline{g_i}(t_1)=\left\{ \begin{array}{ll} \lambda ^2 \, t_1^2, &{} \text {if } |t_1 |\le q_i,\\ \lambda ^2 \, (2\, q_i |t_1 |-q_i^2), &{} \text {if } |t_1 |\ge q_i, \qquad i=1,2 \end{array} \right. \nonumber \\ \end{aligned}$$

(13)

(see also [60]). Moreover, we find an approximation, convex concerning the variable $t_2$, of the function $\varphi$ defined in (7), in the following way:

$$\begin{aligned} \overline{\varphi }(t_1, t_2)= & {} \dfrac{{\overline{t}}^2-t_2^2}{{\overline{t}}^2} \, \overline{g_1}(t_1) + \dfrac{t_2^2}{{\overline{t}}^2} \, \overline{g_2}(t_1) = \overline{g_1}(t_1) \nonumber \\{} & {} + \dfrac{t_2^2}{{\overline{t}}^2} \, (\overline{g_2}(t_1) - \overline{g_1}(t_1)), \end{aligned}$$

(14)

where ${\overline{t}}$ is the maximum value that a finite difference operator can assume (${\overline{t}}=2^k \cdot \sqrt{2} \cdot 255$, $k=1,2,3$, for the light intensity of the images in the range [0,255]). It is easy to check that $g_2 - g_1$ is convex since $0< q_1 < q_2$.

We recall that $t_1=N^k_c \textbf{x}$, $t_2=N^k_{p_k(c)} \textbf{x}$ in the second term of the right hand of (4), and $t_1=V^k_c \textbf{x}$, $t_2=V^k_{p_k(c)} \textbf{x}$ in the third term of the right hand of (4). Let us fix $k \in \{1,2,3\}$ and $c \in C_k$, and choose $\Xi ^k_c \in \{N^k_c,V^k_c\}$. So, $t_1$ is a function of $\textbf{x}$, and $t_1(\textbf{x})= \Xi ^k_c(\textbf{x})$. In particular, $t_1$ depends only on the variables involved in the clique c. Analogously, $t_2(\textbf{x})= \Xi ^k_{p_k(c)}(\textbf{x})$ depends only on the variables involved in the clique $p_k(c)$. Note that the function $\overline{\varphi }$ defined in (14) is componentwise convex. However, the function $\overline{\Phi }(\textbf{x})=\overline{\varphi } (t_1(\textbf{x}),t_2(\textbf{x}))$ is not componentwise convex concerning the components of the vector $\textbf{x} \in {{\mathbb {R}}}^{3nm}$. This is because $c \cap p_k(c) \ne \emptyset$. However, if we choose $t_2(\textbf{x})= \Xi ^k_{\pi _k(c)}(\textbf{x})$ instead of $\Xi ^k_{p_k(c)}(\textbf{x})$, then the function $t_2$ is componentwise convex with respect to $\textbf{x}$, as shown in Appendix D, since $c \cap \pi _k(c)=\emptyset$. Thus, to define the family of the approximations for the algorithm GCNC, we proceed as follows.

If $p \in [0,1]$, put

$$\begin{aligned} E^{(p)}(\textbf{x})= & {} \left\| M(\textbf{x}-\textbf{y})\right\| _2^2 +\sum _{k=1}^3\sum _{c\in C_k} \overline{\varphi }\left( N^k_c\textbf{x}, p \, N^k_{p_k(c)}{} \textbf{x} \right. \nonumber \\{} & {} +\left. (1-p) \, N^k_{\pi _k(c)}{} \textbf{x} \right) + + \sum _{k=1}^3\sum _{c\in C_k} \overline{\varphi }\nonumber \\{} & {} \times \left( V^k_c\textbf{x}, p \, V^k_{p_k(c)}{} \textbf{x} + (1-p) \, V^k_{\pi _k(c)}{} \textbf{x} \right) . \end{aligned}$$

(15)

When $p \in [1,2]$, set

$$\begin{aligned} E^{(p)}(\textbf{x})= & {} \left\| M(\textbf{x}-\textbf{y})\right\| _2^2 +\sum _{k=1}^3\sum _{c\in C_k} \varphi ^{(p)}\left( N^k_c\textbf{x}, N^k_{p_k(c)}{} \textbf{x} \right) + \nonumber \\{} & {} +\sum _{k=1}^3\sum _{c\in C_k} \varphi ^{(p)}\left( V^k_c\textbf{x}, V^k_{p_k(c)}{} \textbf{x} \right) , \end{aligned}$$

(16)

where

$$\begin{aligned} \varphi ^{(p)}(t_1,t_2)=(2 - p) \, \overline{\varphi }(t_1, t_2) + (p-1) \, \varphi (t_1,t_2). \end{aligned}$$

(17)

In Appendix B, we will prove that for each $p \in [1,2]$, $\varphi ^{(p)}$ satisfies the duality conditions that guarantee the edge-preserving properties and the inhibition of double edges [14]. Note that, for $p \in [0,1]$, the stabilizer $\varphi (t_1, t_2)$ is equal to $\overline{\varphi }(t_1, t_2)$, and hence fulfils the same properties. Furthermore, in [11], it is proved that the associated line process is non-Boolean. The hidden line elements become Boolean as far as p tends to 2. However, we experimentally observed that, in natural images, graded discontinuities can help prevent the aliasing effect. Thus, in the experiments, we stop the minimization algorithm at a suitable value of p different from 2, as explained in Sect. 5.

4.1 The NL-SOR Algorithm

To minimize each approximation $E^{(p)}$, we use a nonlinear successive over relaxation (NL-SOR) algorithm, which is widely used in the literature (see also [7, 9, 16]). The NL-SOR is defined as in Algorithm 2, where $\epsilon >0$ is a fixed threshold, $\omega >0$ is the accelerator parameter,

$$\begin{aligned} \displaystyle { T > \max _{\begin{array}{c} i=1,2,\ldots , nm\\ e = r,g,b \end{array}} \, \max _\textbf{x} \, \left\{ \dfrac{\partial ^2_+ E^{(p)} (\textbf{x})}{(\partial x_i^{(e)})^2}, \dfrac{\partial ^2_- E^{(p)} (\textbf{x})}{(\partial x_i^{(e)})^2} \right\} }, \end{aligned}$$

and the symbols $\partial ^2_+$ and $\partial ^2_-$ denote the right and left second partial derivatives, respectively.

Algorithm 2

NL-SOR algorithm

In our experimental results, we have set $\omega =0.2$ and $\epsilon =3 \sqrt{nm}$, where $n\times m$ is the dimension of the involved image. We chose these values after a long experimental phase, where we monitored the trend of the energy function in each step of the algorithm to obtain high-performance results without excessive computational time.

In [16, Theorem 2], the convergence of the algorithm is proved when $E^{(p)}$ is strictly convex and of class $C^2$. However, such a theorem cannot be applied to our setting since our first approximation is componentwise convex and $C^1$ but neither strictly convex nor $C^2$. Thus, in Appendix C, we propose an extension of [16, Theorem 2] to prove that in our case, when $p=0$, the algorithm stops in correspondence with a stationary point. Note that for $p\ne 0$, we have no guarantee of converging to a stationary point of $E^{(p)}$ using the NL-SOR algorithm. However, the experimental results confirm that excellent solutions can still be achieved (see Sect. 5).

5 Experimental Results

In assessing the efficacy of the proposed demosaicing algorithm, we analyze diverse color image datasets, encompassing the Kodak image dataset [43], the McMaster image dataset [75], the Microsoft Demosaicing Canon Dataset [38], and the Microsoft Demosaicing Panasonic Dataset [38]. The Kodak dataset comprises 24 comprehensive full-color images sized at $512 \times 768$ pixels each. Conversely, the McMaster image dataset features 18 images with dimensions of $500 \times 500$ pixels, derived from original high-resolution images sized at $2310 \times 1814$ pixels. Renowned for their widespread adoption as benchmark datasets within demosaicing and various other color image processing domains, the Kodak and McMaster datasets serve as pivotal standards.

Table 1 Parameters used in the noiseless case for weighting the intra-channel smoothness measure (5)

Table 2 Parameters used in the noiseless case for weighting the inter-channel correlation (6)

Table 3 CPSNRs for noiseless Kodak images

Moreover, the Microsoft Demosaicing Dataset Canon encompasses 57 images at $210 \times 318$ pixels, while the Microsoft Demosaicing Dataset Panasonic comprises 500 images of identical dimensions. These raw images originate from distinct camera sources, namely a Canon EOS 550D and a Panasonic Lumix DMC-LX3, respectively. Such datasets collectively facilitate a comprehensive evaluation of the proposed demosaicing algorithm across varied imaging scenarios and devices.

We implement the proposed algorithm in C language and run it in a Linux Ubuntu environment on a computer with an i5-9400F processor at 2.90 GHz. However, minimizing a non-convex function is a complex task; therefore, high computational costs are still necessary. For example, in the case of the Kodak set, the proposed algorithm has an average computation time of 12.493 min for each image.

The free parameters $\lambda$, $\alpha$, and $\varepsilon$ appearing in our energy function have been calibrated on the images of the Kodak dataset and then used for the other datasets. We employed a trial-and-error strategy to look for the parameters that give the best average color peak signal-to-noise ratio (CPSNR) on all the 24 images. The CPSNR quality index, a key metric in our calibration process, is defined as:

$$\begin{aligned} CPSNR=10\log _{10} \left( \dfrac{255^2}{MSE} \right) , \end{aligned}$$

(18)

Table 4 CPSNRs for McMaster noiseless images

Table 5 CPSNRs for Microsoft images—noiseless

Table 6 Regularization parameters depending of the noise standard deviation $\sigma$

where MSE is the mean square error between the original image and the demosaiced one that, for color images, is defined as the arithmetic average of the mean square errors on the three channels.

We chose to increase p with a step of 0.01. Note that, for $p=2$, the discontinuities present in the reconstructed image are Boolean, whereas, for $p\ne 2$, we have reconstructions with non-Boolean line variables, as stated in the duality theorem given in [15].

Reconstructions obtained with Boolean lines are often disturbed by the aliasing effect, so that it may be preferable to stop the algorithm at a value ${\bar{p}}$ smaller than 2. The optimal value of ${\bar{p}}$ has been set as the one for which the sum of the MSEs of all the 24 Kodak images reconstructed with that p is minimum. Calling $\eta _j(p)$ the MSE between the ideal image and the minimizer of the approximated energy function for the $j-th$ image at a given p, it is:

$$\begin{aligned} {{\bar{p}}}= \arg \min _p \left\{ \sum _{j=1}^{24}\eta _j(p) \right\} . \end{aligned}$$

(19)

The proposed algorithm has been applied to both noiseless and noisy images, and the performance of the method has been compared with some of the most popular and best-performing methods in the literature.

5.1 Noiseless Images

In the energy E (4), stabilizer $\varphi$ depends on the free parameters $\lambda$, $\alpha$ and $\varepsilon$. In the second addend of the right hand of (4), we used $\varphi$ with the free parameters reported in Table 1, which we found to be the best for noiseless mosaiced images.

In the third term of the right hand of (4), the best free parameters for stabilizer $\varphi$ to be used in the noiseless case are indicated in Table 2.

Given all these free parameters, we found ${{\bar{p}}} = 1.42$.

For the noiseless Kodak dataset, we compared the proposed method with the algorithms in [29, 33, 50, 1, 2, 19, 54] [58, 41], and [59]. In particular, we used the source code available on the web to obtain the results of the algorithm in [59]. The results obtained on the Kodak dataset are reported in Table 3. As usual with the Kodak images, we removed the 3 pixels wide external frame to compute the errors. Our method exhibits the highest CPSNR in more images than the other methods (the highest CPSNR is highlighted in boldface for each image). Furthermore, the proposed method achieves the best result on average.

Table 7 CPSNRs for noisy Kodak images, $\sigma =16$

For the noiseless McMaster dataset, a comparison has been made with the algorithms in [17, 29, 33, 41, 75], and [59], respectively. In particular, to collect the results of the algorithm in [29], we used the original MATLAB code provided by the authors, and for the results of the algorithms in [41] and [59], we used the source code available on the web. The results obtained are reported in Table 4. The proposed method achieved the best result in more than $44\%$ of cases. However, this time, the method presented in [59] is slightly better on average. We thus used the Microsoft Demosaicing Canon Dataset [38] and the Microsoft Demosaicing Panasonic Dataset [38] to further compare our method with the one proposed in [59]. Table 5 presents the obtained average results on both datasets; the proposed algorithm always obtains the best results.

5.2 Noisy Images

In a second set of experiments, we considered noisy images corrupted by independent Gaussian noise, with zero mean and different standard deviation values $\sigma$. This time, the best free parameters for stabilizer $\varphi$ to be used in the second and third addend of the right hand of (4) are shown in Table 6 .

As done for the noiseless images, for each value of the noise variance, the suitable value ${\bar{p}}$ for stopping the algorithm has been determined according to the criterion established in (19). The following empirical law that relates ${\bar{p}}$ to $\sigma$ has also been found:

$$\begin{aligned} {\bar{p}}(\sigma )=\dfrac{3}{40}\sigma +\dfrac{4}{5}. \end{aligned}$$

(20)

In the noisy case, we compared our method with the algorithms in [29] and [34] by using the original MATLAB code provided by the authors and with the algorithms in [20, 76], and [55], by using the source codes available in the authors’ web pages. The CPSNR values computed for the case $\sigma =16$ on the Kodak dataset are shown in Table 7.

Although the performance of our method is still satisfactory, this time, the method in [20] is slightly superior.

We then computed another quality index, sometimes used in the demosaicing problem, i.e., the S-CIELAB metric. This metric indicates the percentage of color distortion between two images and accounts for the spatial-color sensitivity of the human eye [73] [37]. Since it returns a pixel-by-pixel matrix of errors, we assumed the mean of the S-CIELAB matrix coefficients as the representative error index for the entire image. The results obtained for the case $\sigma =16$ on the Kodak dataset, along with the results of the most performing among the methods used for comparison, are shown in Table 8.

The results in this case are excellent. The situation is even better when the noisy mosaiced McMaster images are processed. The CPSNR results obtained for the same amount of noise ($\sigma =16$), along with the results of the most performing methods used for comparison on the Kodak dataset, are shown in Table 9.

It is apparent that this time, our method outperforms the other, with much higher values of CPSNR, which are only slightly lower than those we obtained in the noiseless case for the same dataset. This excellent performance can be ascribed once again to our very fine modeling of natural images in terms of local variations inside and between the color channels.

Table 8 S-CIELAB errors for noisy Kodak images, $\sigma =16$

Table 9 CPSNRs for noisy McMaster images, $\sigma =16$

Furthermore, in Table 10, it is possible to observe the excellent behavior of the proposed method on the two Microsoft datasets, compared with the method in [59] and for different values of the standard deviation $\sigma$ of the noise corrupting the data.

6 Conclusion

We approached the joint demosaicing and denoising of color images within a regularization framework, irrespective of the CFA employed to generate the data. A central feature of our method is adopting local image smoothness models that implicitly account for “soft” edges at low- and high-frequency levels. Soft edges are essential for preventing the aliasing effect and preserving the fine details in the image, especially when noise is present. In our method, this is achieved by using stabilizers possessing two main features: on the one hand, we include derivatives up to the third degree; on the other hand, these derivatives are weighted through non-quadratic functions. A duality theorem ensures that the implicitly addressed edge fields are equivalent to geometrically consistent image discontinuities at each cycle of the iterative algorithm employed to find the regularized solution.

The experimentation over the Kodak 24-image dataset, the McMaster (IMAX) 18-image dataset, the Microsoft Demosaicing Canon 57-image dataset, and the Microsoft Demosaicing Panasonic 500-image dataset under the Bayer CFA, with a high level of noise, demonstrates the good performance of our method against some of the best-performing demosaicing algorithms proposed so far.

Future developments involve experimenting with the algorithm on data from other CFAs and including the filter blurs that are known in this kind of application. For this latter issue, the extension of the algorithm is straightforward, since the data term is still convex, so the same approximations for the energy function can also be exploited.

Table 10 CPSNRs for Microsoft images—noisy

Appendices

Appendix A: Geometry of the Cliques and Expression of the Associated Finite Differences

The stabilizers used in this paper are functions of the finite differences $D_c^k$ of order k applied to sets c consisting of adjacent pixels. We call clique of order k the set of pixels on which the finite difference of order k is well defined. We take $k=1,2,3$ in order to reconstruct the finest details in images. Figures 4, 5 and 6 show the geometry of the sets c for the three orders of finite differences, respectively. As we can see, the cliques can be classified as vertical (Figs. 4 (a), 5 (a), 6 (a) ), horizontal (Figs. 4 (b), 5 (c), 6 (d) ), and mixed (Figs. 5 (b), 6 (b) and (c) ).

The vertical cliques consist of the following pixels:

$$\begin{aligned} c= & {} \{(i,j), (i+1,j), \ldots , (i+k,j) \}, \nonumber \\ i= & {} 1, \ldots , n-k, \, j=1,\ldots , m, \, k=1,2,3, \end{aligned}$$

(A1)

while the horizontal cliques have the form

$$\begin{aligned} c= & {} \{(i,j), (i,j+1), \ldots , (i,j+k) \}, \nonumber \\ i= & {} 1, \ldots , n, \, j=1,\ldots , m-k, \, k=1,2,3. \end{aligned}$$

(A2)

Fig. 4

Geometry of the sets c for $k=1$

Fig. 5

Geometry of the sets c for $k=2$

Fig. 6

Geometry of the sets c for $k=3$

Let us now describe how finite differences are computed at a generic clique c for a generic color channel $\textbf{x}^{(e)}$, $e \in \{r,g,b\}$. When $k=1$, as it is seen in Fig. 4, we have two different kinds of finite difference operators, associated with a horizontal and a vertical finite difference, given by

$$\begin{aligned} D^1_{c}{} \textbf{x}^{(e)}=\left\{ \begin{array}{lll} x^{(e)}_{(i,j)}- x^{(e)}_{(i+1,j)} &{} \text{ in }\,\, 4 (a); \\ x^{(e)}_{(i,j)}- x^{(e)}_{(i,j+1)} &{} \text{ in }\,\, 4(b), \end{array}\right. \end{aligned}$$

(A3)

respectively. When $k=2$, we have three different kinds of finite difference operators, expressed by

$$\begin{aligned} D^2_{c}{} \textbf{x}^{(e)}=\left\{ \begin{array}{lll} x^{(e)}_{(i,j)}-2 x^{(e)}_{(i+1,j)} +x^{(e)}_{(i+2,j)}\qquad \qquad \!\! \text{ in }\,\, 5 (a); \\ x^{(e)}_{(i,j)}-2 x^{(e)}_{(i,j+1)} +x^{(e)}_{(i,j+2)}\qquad \qquad \!\! \text {in}\,\, 5 (b); \\ x^{(e)}_{(i,j)}-x^{(e)}_{(i+1,j)}- x^{(e)}_{(i,j+1)}+x^{(e)}_{(i+1,j+1)}\\ \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \quad \text{ in }\,\, 5 (c). \end{array}\right. \end{aligned}$$

(A4)

When $k=3$, we get four different kinds of finite difference operators, given by

$$\begin{aligned} D^3_{c} \textbf{x}^{(e)} =\left\{ \begin{array}{lll} x^{(e)}_{(i,j)} -3x^{(e)}_{(i+1,j)} +3x^{(e)}_{(i+2,j)} -x^{(e)}_{(i+3,j)} \nonumber \\ \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \quad \ \, \text{ in }\, 6 (a); \nonumber \\ x^{(e)}_{(i,j)}-3 x^{(e)}_{(i,j+1)} +3x^{(e)}_{(i,j+2)}-x^{(e)}_{(i,j+3)}\\ \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \quad \ \, \text{ in }\, 6 (b); \nonumber \\ x^{(e)}_{(i,j)}-2 x^{(e)}_{(i+1,j)} +x^{(e)}_{(i+2,j)} -x^{(e)}_{(i,j+1)}\nonumber \\ \quad +2 x^{(e)}_{(i+1,j+1)} -x^{(e)}_{(i+2,j+1)} \qquad \!\!\qquad \!\! \text{ in }\,\, 6 (c); \\ x^{(e)}_{(i,j)} -2x^{(e)}_{(i,j+1)} + x^{(e)}_{(i,j+2)}- x^{(e)}_{(i+1,j)} \nonumber \\ \quad + 2x^{(e)}_{(i+1,j+1)} -x^{(e)}_{(i+1,j+2)} \!\!\!\!\qquad \qquad \text{ in }\,\, 6 (d). \end{array}\right. \\ \end{aligned}$$

(A5)

Let us introduce the concept of adjacent clique of order k, which defines the non-parallelism constraint. Given a vertical clique

$c=\{(i,j), (i+1,j), \ldots , (i+k,j) \}$,   $i=k+1, \ldots , n-k$,   $j=1,\ldots , m$,   $k=1,2,3$,

we define its preceding clique $p_k(c)$ as follows:

$$\begin{aligned} p_k(c)=\{(i-k,j), (i-k+1,j), \ldots , (i,j) \}. \end{aligned}$$

When $i=k+2, \ldots , n-k$,   $j=1,\ldots , m$,   $k=1,2,3$, a good approximation of $p_k(c)$ used to construct our approximating functions is given by

$$\begin{aligned} \pi _k(c)=\{(i-k-1,j), (i-k,j), \ldots , (i-1,j) \}. \end{aligned}$$

We define $\pi _k(c)$ in such a way that $c \cap \pi _k(c)= \emptyset$. This will be useful to find the family of approximations of the energy function in Subsection 3.2.

If c is a horizontal clique, $c=\{(i,j), (i,j+1), \ldots , (i,j+k) \}$,   $i=1, \ldots , n$,   $j=k+1,\ldots , m-k$,   $k=1,2,3$, then its preceding clique $p_k(c)$ is defined by

$$\begin{aligned} p_k(c)=\{(i,j-k), (i,j-k+1), \ldots , (i,j) \}. \end{aligned}$$

When $i=1, \ldots , n$,   $j=k+2,\ldots , m-k$,   $k=1,2,3$, a good approximation of $p_k(c)$ is

$$\begin{aligned} \pi _k(c)=\{(i,j-k-1), (i,j-k), \ldots , (i,j-1) \}. \end{aligned}$$

For mixed cliques and cliques on the board of the image, $p_k(c)$ and $\pi _k(c)$ are considered not to be defined.

Appendix B: Duality Conditions on the Stabilizer

In order for a stabilizer $\varphi$ to be edge-preserving and that the non-parallelism constraint on the implicit line process is satisfied, we require that the hypotheses of the following theorem are satisfied (see [14]):

Theorem 1

For every $p \in [1,2]$, let

$$\begin{aligned} \varphi ^{(p)}(t_1,t_2)= & {} (2 - p) \, \overline{\varphi }(t_1, t_2) + (p-1) \, \varphi (t_1,t_2), \nonumber \\{} & {} t_1 \in {\mathbb {R}}, t_2 \in [-{\overline{t}}, {\overline{t}}], \end{aligned}$$

(B6)

where ${\overline{t}}= 2^k \cdot \sqrt{2} \cdot 255$, $k=1,2,3$, for light intensity of the images in the range [0, 255], is the maximum value which the variable $t_2$ can assume, $\overline{\varphi }$ and $\varphi$ are as in (14) and (7), respectively. Then, $\varphi ^{(p)}$ satisfies the following conditions:

H1):

for every $t_2 \in [-{\overline{t}}, {\overline{t}}]$, the function $\varphi _{t_2}: {\mathbb {R}} \rightarrow {\mathbb {R}} \cup \{ - \infty \}$ defined by $\varphi _{t_2}(t_1)=\varphi (t_1, t_2)$ is upper semicontinuous and even on ${\mathbb {R}}$, and $\varphi _{t_2}(0)\in {\mathbb {R}}$;

H2):

for each $t_2 \in [-{\overline{t}}, {\overline{t}}]$, the function $\psi _{t_2}: {\mathbb {R}} \rightarrow {\mathbb {R}} \cup \{ - \infty \}$ defined by

$$\begin{aligned} \psi _{t_2}(t_1)= \left\{ \begin{array}{ll}\varphi (\sqrt{t_1}, t_2), &{} \text { if } t_1 \ge 0,\\ -\infty , &{} \text { if } t_1 < 0, \end{array}\right. \end{aligned}$$

is concave on ${\mathbb {R}}^+_0$;

H3):

$\varphi _{t_2}$ is non-decreasing on ${\mathbb {R}}^+_0$ for every $t_2 \in [-{\overline{t}}, {\overline{t}}]$;

H4):

$\displaystyle {\lim _{t_1\rightarrow + \infty } \dfrac{\psi _{t_2}(t_1)}{t_1} =0}$ for each $t_2 \in [-{\overline{t}}, {\overline{t}}]$,

H5):

there exists at least a real number $t_1$ such that the function $\varphi _{t_1}(t_2)=\varphi (t_1, t_2)$ is not constant on $[-{\overline{t}}, {\overline{t}}]$, and $\varphi _{t_1}$ is even on $[-{\overline{t}}, {\overline{t}}]$ and non-decreasing on $[0, {\overline{t}}]$ for every $t_1 \in {\mathbb {R}}^+_0$.

Proof

We begin by proving that the function $\overline{\varphi }$ defined in (14) satisfies Hj), $j=1, \ldots , 4$.

It is readily seen that $\overline{\varphi }$ fulfills H1).

Now we prove H2).

For $i=1,2$ and $t_1 \in {\mathbb {R}}^+_0$, set

$$\begin{aligned} \overline{f_i}(t_1)=\overline{g_i}(\sqrt{t_1})=\left\{ \begin{array}{ll} \lambda ^2 \, t_1, &{} \text {if } 0 \le t_1\le q_i^2,\\ \lambda ^2 \, (2\, q_i \sqrt{t_1} -q_i^2), &{} \text {if } t_1\ge q_i^2. \end{array} \right. \nonumber \\ \end{aligned}$$

(B7)

We have

$$\begin{aligned} {\overline{\varphi }}_{t_2}(\sqrt{t_1})= {\overline{\psi }}_{t_2}(t_1)= \dfrac{{\overline{t}}^2-t_2^2}{{\overline{t}}^2} \, \overline{f_1}(t_1) + \dfrac{t_2^2}{{\overline{t}}^2} \, \overline{f_2}(t_1), \end{aligned}$$

and hence

$$\begin{aligned} {\overline{f_i}}^{\prime }(t_1)=\left\{ \begin{array}{ll} \lambda ^2, &{} \text {if } 0 \le t_1 \le q_i^2,\\ \lambda ^2 \, q_i \,{t_1}^{-1/2}, &{} \text {if } t_1\ge q_i^2; \end{array} \right. \\ {\overline{f_i}}^{\prime \prime }(t_1)=\left\{ \begin{array}{ll} 0, &{} \text {if } 0 \le t_1 < q_i^2,\\ \displaystyle { -\dfrac{1}{2} \lambda ^2 \, q_i \,{t_1}^{-3/2},} &{} \text {if } t_1 > q_i^2. \end{array} \right. \end{aligned}$$

Let $\displaystyle {\gamma _{t_2}= \dfrac{{\overline{t}}^2-t_2^2}{{\overline{t}}^2}}$. Then, $1-\gamma _{t_2}=\dfrac{t_2^2}{{\overline{t}}^2}$.

It is not difficult to check that $0 \le \gamma _{t_2} \le 1$, since $|t_2 |\le {\overline{t}}$. Thus, for every $t_1 \in {\mathbb {R}}^+_0$ and $t_1 \ne q_1^2$, $t_1 \ne q_2^2$, $t_2 \in [-{\overline{t}}, {\overline{t}}]$, we have

$$\begin{aligned} {{\overline{\psi }}_{t_2}}^{\prime }(t_1)= \gamma _{t_2} \, {\overline{f_1}}^{\prime }(t_1) + (1-\gamma _{t_2}) \, {\overline{f_2}}^{\prime }(t_1) \ge 0. \end{aligned}$$

(B8)

$$\begin{aligned} {{\overline{\psi }}_{t_2}}^{\prime \prime }(t_1)= \gamma _{t_2} \, {\overline{f_1}}^{\prime \prime }(t_1) + (1-\gamma _{t_2}) \, {\overline{f_2}}^{\prime \prime }(t_1) \le 0. \end{aligned}$$

Observe that the inequality in (B8) will be useful to prove H3). Since ${\overline{\psi }}_{t_2}$ is of class $C^1$ on its domain (indeed, it is a composition of $C^1$ functions), then it is concave on ${\mathbb {R}}_0^+$ for all $t_2 \in [-{\overline{t}}, {\overline{t}}]$. So, $\overline{\varphi }$ satisfies condition H2).

Now we prove H3). From (B8), it follows that ${\overline{\psi }}_{t_2}$ is non-decreasing on ${\mathbb {R}}^+_0$, and hence so is ${\overline{\varphi }}_{t_2}$.

Thus we get

$$\begin{aligned} {{\overline{\varphi }}_{t_2}}^{\prime }(t_1)= \gamma _{t_2} \, \overline{{g_1}}^{\prime }(t_1) + (1-\gamma _{t_2}) \, \overline{{g_2}}^{\prime } (t_1)\ge 0 \end{aligned}$$

for each $t_1 \in {\mathbb {R}}^+_0$. Thus, H3) holds.

Now we show that $\overline{\varphi }$ fulfils H4). Indeed, we have

$$\begin{aligned} \lim _{t_1 \rightarrow + \infty } \dfrac{\overline{f_i}(t_1)}{t_1}= \lim _{t_1 \rightarrow + \infty } \dfrac{\lambda ^2 \, (2\, q_i \sqrt{t_1} - q_i^2)}{t_1}=0 \quad (i=1,2), \end{aligned}$$

and hence

$$\begin{aligned} \lim _{t_1 \rightarrow + \infty } \dfrac{{\overline{\psi }}_{t_2}(t_1)}{t_1}= 0 \quad \text { for every } t_2 \in [-{\overline{t}}, {\overline{t}}]. \end{aligned}$$

Finally, we prove that $\overline{\varphi }$ satisfies H5).

Take $t_1=q_2$. For each $t_2 \in [-{\overline{t}}, {\overline{t}}]$, it is

$$\begin{aligned} \overline{\varphi }(q_2, t_2)=\dfrac{{\overline{t}}^2-t_2^2}{{\overline{t}}^2} \, \lambda ^2 \, ( 2 \, q_1 \, q_2 - q_1^2) + \dfrac{t_2^2}{{\overline{t}}^2} \, \lambda ^2 \, q_2^2, \end{aligned}$$

and hence $\overline{\varphi }(q_2,0) =\lambda ^2 \, ( 2 \, q_1 \, q_2 - q_1^2)$, $\overline{\varphi }(q_2,{\overline{t}})=\lambda ^2 \, q_2^2$. We claim that $\overline{\varphi }(q_2,0) \ne \overline{\varphi }(q_2,{\overline{t}})$. If not, then we would have $2 \, q_1 \, q_2 - q_1^2 = q_2^2$, and hence $0=q_2^2 - 2 \, q_1 \, q_2 + q_1^2 = (q_1 - q_2)^2$, that is $q_1=q_2$, which is absurd since we know that $0< q_1 < q_2$. Therefore, the function $t_2 \mapsto \overline{\varphi }(q_2, t_2)$ is not constant, and hence, the first property of H5) is satisfied.

Moreover, it is easy to see that ${\overline{\varphi }}_{t_1}$ is even on ${\mathbb {R}}$ for each $t_1 \in {\mathbb {R}}$.

From (13), it is not difficult to deduce that $\overline{g_2}(t_1) - \overline{g_1} (t_1) \ge 0$ for all $t_1 \ge 0$.

We get

$$\begin{aligned} \dfrac{d \overline{\varphi }_{t_1}}{d t_2} (t_2)= & {} -\dfrac{2 \, t_2}{{\overline{t}}^2} \, {\overline{g_1}}(t_1) + \dfrac{2 \, t_2}{{\overline{t}}^2} \, {\overline{g_2}}(t_1)\nonumber \\= & {} \dfrac{2 \, t_2}{{\overline{t}}^2} \, ({\overline{g_2}}(t_1) - {\overline{g_1}}(t_1)) \ge 0 \end{aligned}$$

(B9)

for any $t_1 \in {\mathbb {R}}^+_0$. Hence, the function $\overline{\varphi }_{t_1}$ is non-decreasing on ${\mathbb {R}}^+_0$ for every $t_1 \in {\mathbb {R}}^+_0$. Thus, H5) is proved.

Now we prove that for $i=1,2$, the function $\varphi$ defined in (7) satisfies conditions Hj), $j=1, \ldots , 4$.

It is easy to see that, by construction, H1) holds.

We now prove H2). We begin with the case when $|t_2 |\le s$ or $t_2 \ge \zeta$ in (7). For $i=1,2$, set

$$\begin{aligned} f_i(t_1)=g_i(\sqrt{t_1}) = \left\{ \begin{array}{ll} \lambda ^{2}t_1, &{} \text { if } 0 \le t_1 \le q_i^2,\\ \\ \alpha _i -\dfrac{\tau }{2}(\sqrt{t_1}-r_i)^{2}, &{} \text { if } \, q_i^2\le t_1\le r_i^2, \\ \\ \alpha _i, &{} \text { if } \,t_1 \ge r_i^2. \end{array} \right. \end{aligned}$$

(B10)

We have

$$\begin{aligned} \varphi (\sqrt{t_1},t_2) =\left\{ \begin{array}{ll} f_1(t_1), \qquad \qquad \qquad \qquad \qquad \qquad \text { if } |t_2 |\le s,\\ \Bigl ( 1- \dfrac{2(|t_2 |-s)^2}{(\zeta -s)^2} \Bigl ) \, f_1(t_1) + \frac{2(|t_2 |-s)^2}{(\zeta -s)^2} \, f_2(t_1),\\ \qquad \qquad \qquad \qquad \qquad \qquad \ \text { if } s< |t_2 |\le \frac{\zeta +s}{2},\\ \frac{2(|t_2 |-\zeta )^2}{(\zeta -s)^2} \, f_1(t_1) + \Bigl ( 1- \frac{2(|t_2 |-\zeta )^2}{(\zeta -s)^2} \Bigl ) \, f_2(t_1), \\ \qquad \qquad \qquad \qquad \qquad \qquad \ \text { if } \frac{\zeta +s}{2}< |t_2 |<\zeta ,\\ f_2(t_1), \qquad \qquad \qquad \qquad \qquad \qquad \text { if } |t_2 |\ge \zeta . \end{array}\right. \end{aligned}$$

(B11)

We claim that $f_i$ is non-decreasing and concave on ${\mathbb {R}}^+_0$. We get

$$\begin{aligned} {f_i}^{\prime }(t_1) = \left\{ \begin{array}{ll} \lambda ^{2}, &{} \text { if } 0 \le t_1 \le q_i^2,\\ \\ \alpha _i -\dfrac{\tau }{2}\Bigr (1- \dfrac{r_i}{\sqrt{t_1}} \Bigl ), &{} \text { if } \, q_i^2\le t_1\le r_i^2, \\ \\ 0, &{} \text { if } \,t_1 \ge r_i^2. \end{array} \right. \end{aligned}$$

(B12)

Note that $f_i$ is $C^1$, since it is a composition of functions of class $C^1$. Moreover, we have

$$\begin{aligned} {f_i}^{\prime \prime }(t_1) = \left\{ \begin{array}{ll} 0, &{} \text { if } 0 \le t_1<q_i^2,\\ \\ -\dfrac{\tau \, r_i}{4 \, \sqrt{t_1^3}}, &{} \text { if } \, q_i^2< t_1 < r_i^2, \\ \\ 0, &{} \text { if } \,t_1>r_i^2. \end{array} \right. \end{aligned}$$

(B13)

From this, we deduce that $\varphi$ fulfils H2), at least when $|t_2 |\le s$ or $|t_2 |\ge \zeta$.

Now, we examine the case

$$\begin{aligned} s< |t_2 |\le \dfrac{\zeta +s}{2}. \end{aligned}$$

(B14)

We have

$$\begin{aligned} \psi _{t_2}^{\prime }(t_1)= & {} \Bigl ( 1- \dfrac{2(|t_2 |-s)^2}{(\zeta -s)^2} \Bigl ) \, f_1^{\prime }(t_1) + \dfrac{2(|t_2 |-s)^2}{(\zeta -s)^2} \, f_2^{\prime }(t_1),\nonumber \\ \end{aligned}$$

(B15)

$$\begin{aligned} \psi _{t_2}^{\prime \prime }(t_1)= & {} \Bigl ( 1- \dfrac{2(|t_2 |-s)^2}{(\zeta -s)^2} \Bigl ) \, f_1^{\prime \prime }(t_1) + \dfrac{2(|t_2 |-s)^2}{(\zeta -s)^2} \, f_2^{\prime \prime }(t_1).\nonumber \\ \end{aligned}$$

(B16)

Observe that $\dfrac{2(|t_2 |-s)^2}{(\zeta -s)^2}\ge 0$. Now, we claim that $\dfrac{2(|t_2 |-s)^2}{(\zeta -s)^2}\le 1$. Indeed, since $s< |t_2 |\le \dfrac{\zeta +s}{2}$, then $0<|t_2 |-s \le \dfrac{\zeta -s}{2} < \dfrac{\zeta -s}{\sqrt{2}}$, and hence $(|t_2 |-s)^2 \le \dfrac{(\zeta -s)^2}{2}$, getting the claim. Therefore, $1- \dfrac{2(|t_2 |-s)^2}{(\zeta -s)^2} \ge 0$. From this, since $f_i^{\prime }(t_1) \ge 0$ for every $t_1 \ge 0$ and $f_i^{\prime \prime }(t_1) \le 0$ for each $t_1 {\mathbb {R}}^+_0$, $t_1 \ne q_1$, $t_1 \ne q_2$, in the case (B14) we obtain

$$\begin{aligned}{} & {} \psi _{t_2}^{\prime }(t_1) \ge 0 \text { for every } t_1 \in {\mathbb {R}}^+_0, \end{aligned}$$

(B17)

$$\begin{aligned}{} & {} \psi _{t_2}^{\prime \prime }(t_1) \le 0 \text { for any } t_1 \in {\mathbb {R}}^+_0, \, t_1 \ne q_1, \, t_1 \ne q_2. \end{aligned}$$

(B18)

Note that the inequality in (B17) will be useful to prove H3).

From (B18), taking into account the continuity of $\psi _{t_2}$, we deduce that $\varphi$ satisfies H2) also in the case (B14).

Now, we consider the case

$$\begin{aligned} \dfrac{\zeta +s}{2}< |t_2 |< \zeta . \end{aligned}$$

(B19)

We get

$$\begin{aligned} \psi _{t_2}^{\prime }(t_1)= & {} \dfrac{2(|t_2 |-\zeta )^2}{(\zeta -s)^2} \, f_1^{\prime }(t_1) + \Bigl ( 1- \dfrac{2(|t_2 |-\zeta )^2}{(\zeta -s)^2} \Bigl ) \, f_2^{\prime }(t_1),\nonumber \\ \end{aligned}$$

(B20)

$$\begin{aligned} \psi _{t_2}^{\prime \prime }(t_1)= & {} \dfrac{2(|t_2 |-\zeta )^2}{(\zeta -s)^2} \, f_1^{\prime \prime }(t_1) + \Bigl ( 1- \dfrac{2(|t_2 |-\zeta )^2}{(\zeta -s)^2} \Bigl ) \, f_2^{\prime \prime }(t_1).\nonumber \\ \end{aligned}$$

(B21)

Note that $\dfrac{2(|t_2 |-\zeta )^2}{(\zeta -s)^2}\ge 0$. Now we claim that $\dfrac{2(|t_2 |-\zeta )^2}{(\zeta -s)^2}\le 1$. Indeed, as $\dfrac{\zeta +s}{2}< |t_2 |< \zeta$, then $0<\zeta - |t_2 |\le \dfrac{\zeta -s}{2} < \dfrac{\zeta -s}{\sqrt{2}}$, and so $(|t_2 |-\zeta )^2 \le \dfrac{(\zeta -s)^2}{2}$, getting the claim. Thus, $1- \dfrac{2(|t_2 |-\zeta )^2}{(\zeta -s)^2} \ge 0$. From this, in the case (B19), analogously as in the case (B14), we obtain

$$\begin{aligned}{} & {} \psi _{t_2}^{\prime }(t_1) \ge 0 \text { for every } t_1 \in {\mathbb {R}}^+_0, \end{aligned}$$

(B22)

$$\begin{aligned}{} & {} \psi _{t_2}^{\prime \prime }(t_1) \le 0 \text { for any } t_1 \in {\mathbb {R}}^+_0, \, t_1 \ne q_1, \, t_1 \ne q_2. \end{aligned}$$

(B23)

From (B23) and thanks to the continuity of $\psi _{t_2}$, it follows that $\varphi$ satisfies H2) also in the case (B19).

Now we prove that $\varphi$ satisfies H3).

First, when $|t_2 |\le s$ or $|t_2 |\ge \zeta$, observe that it is readily seen that $g_i$ is non-decreasing on ${\mathbb {R}}^+_0$ for $i=1,2$. Hence, $\psi _{t_2}$ is non-decreasing on ${\mathbb {R}}^+_0$, and thus H3) holds.

Moreover, when $t_2$ satisfies the case (B14) or the case (B19), from (B17) and (B22) it follows that $\psi _{t_2}$ is non-decreasing on ${\mathbb {R}}^+_0$, and hence $\varphi _{t_2}$ is too. Thus, $\varphi$ fulfils H3).

Now we prove H4). That $\varphi$ satisfies H4) is a consequence of the fact that $\displaystyle {\lim _{t_1\rightarrow + \infty } \dfrac{f_i(t_1)}{t_1} =0}$.

Now we prove H5). Let

$$\begin{aligned} a(t_1)=2 \, \dfrac{g_2 (t_1)-g_1(t_1)}{(\zeta -s)^{2}}, \quad t_1 \in {\mathbb {R}}. \end{aligned}$$

First of all, observe that $\varphi (r_2,s)=g_1(r_2)=\alpha$, $\varphi (r_2,\zeta )=g_2(r_2)=\alpha +\varepsilon \ne \varphi (r_2,s)$. Thus, the function $t_2\mapsto \varphi (r_2, t_2)$ is not constant. Moreover, it is easy to see that the function $\varphi _{t_1}(t_2)$ is even on $[-{\overline{t}}, {\overline{t}}]$ for every $t_1 \in {\mathbb {R}}^+_0$, since it depends on $|t_2 |$. Furthermore, it is not difficult to check that

$$\begin{aligned} g_2(t_1) \ge g_1(t_1) \quad \text {for every } t_1 \in {\mathbb {R}}^+_0. \end{aligned}$$

(B24)

Let $t_2 \in [0, {\overline{t}}]$. We get

$$\begin{aligned} {\varphi _{t_1}}^{\prime }(t_2)= -\dfrac{2 (t_2-s)^2}{(\zeta -s)^2} \, g_1(t_1) + \dfrac{2(t_2-s)^2}{(\zeta -s)^2} \, g_2(t_1) \end{aligned}$$

(B25)

in the case (B14), and

$$\begin{aligned} {\varphi _{t_1}}^{\prime }(t_2)= -\dfrac{2 (t_2-\zeta )^2}{(\zeta -s)^2} \, g_1(t_1) - \dfrac{2(t_2-\zeta )^2}{(\zeta -s)^2} \, g_2(t_1) \end{aligned}$$

(B26)

in the case (B19). From (B24), (B25) and (B26) it follows that $\varphi _{t_1}^{\prime }(t_2)\ge 0$ for each $t_2 \in [0, {\overline{t}}]$. By arbitrariness of $t_1 \in {\mathbb {R}}^+_0$, we deduce that $\varphi$ satisfies H5).

Now, we observe that the functions $\varphi ^{(p)}$, $p \in [0,2]$, satisfy conditions Hj), $j=1, \ldots , 4$, since they are non-negative linear combinations of functions satisfying Hj), $j=1, \ldots , 4$. Since $\overline{\varphi }_{t_1}$ and $\varphi _{t_1}$ are non-decreasing for each $t_1 \in {\mathbb {R}}^+_0$, $\overline{\varphi }_{t_2}$ and $\varphi _{t_2}$ are non-decreasing for every $t_2 \in [-{\overline{t}}, {\overline{t}}]$, and the functions $t_2 \mapsto \overline{\varphi }(q_2,t_2)$, $t_2 \mapsto \varphi (r_2,t_2)$ are not constant, it follows that for every $p \in [0,2]$ there exists at least a $t_1 \in {\mathbb {R}}^+_0$ such that the function $t_2 \mapsto \varphi ^{(p)}(t_1, t_2)$ is not constant. The other properties of H5) hold because the $\varphi ^{(p)}$’s are non-negative linear combinations of functions satisfying H5). $\square$

Appendix C: Convergence of the NL-SOR Algorithm

To minimize each approximation $E^{(p)}$, $p \in [0,2]$, we use the NL-SOR algorithm, described in Subsection 4.1.

We will prove the existence of suitable limit points, which are stationary points of $E^{(0)}$ (in general, they are not minimum points of $E^{(0)})$. In [16, Theorem 2] the convergence of the algorithm is proved when $E^{(0)}$ is strictly convex and of class $C^2$. Such assumptions are too strong for the componentwise convex approximation of the regularization term in our setting because we deal with functions of class $C^1$. So, we give an extension of the theorem under these weaker hypotheses.

First, we state the following technical lemma, whose proof is given in [28, Lemma 2.7.1].

Lemma 2

Let $\phi :[x_0,{\overline{x}}] \rightarrow {\mathbb {R}}$ be convex, of class $C^1$, having both left and right second derivative on $[x_0,{\overline{x}}]$. Suppose that $\phi$ is second differentiable on $[x_0, {\overline{x}}] \setminus P$, where $P=\{ x_j: j=1, \ldots , N \}$, with $x_0< x_1< \ldots< x_N < {\overline{x}}$.

Then, for every $x \in [x_0,{\overline{x}}]$ there exist $\xi \in ]x_0,x[$ and $\mu \ge 0$, such that

$$\begin{aligned} \mu \in I_{\xi }=[\min \{ \phi ^{\prime \prime }_{-}(\xi ), \phi ^{\prime \prime }_{+}(\xi )\}, \max \{\phi ^{\prime \prime }_{-}(\xi ), \phi ^{\prime \prime }_{+}(\xi )\}]\nonumber \\ \end{aligned}$$

(C27)

and

$$\begin{aligned} \phi (x)=\phi (x_0) + (x-x_0) \phi ^{\prime }(x_0)+ \dfrac{(x-x_0)^2}{2} \, \mu . \end{aligned}$$

(C28)

Observe that when $p=0$, the NL-SOR can be formulated as in Algorithm 3. At the iterate $l \in {\mathbb {N}}$, fixed $i \in {1,2, \ldots , 3\,m}$ and $e \in \{r,g,b\}$, the vector $\textbf{x}^{(l, i, e)}$ is defined by

$$\begin{aligned} (x^{(l, i, e)})_j^{(q)}= \left\{ \begin{array}{ll} (x^{\text {prec}(l, i, e)})_j^{(q)} \qquad \qquad \qquad \text {if } i \ne j \text { or } q \ne e, \\ (x^{\text {prec}(l, i, e)})_j^{(q)} - \dfrac{\omega }{T} \, \dfrac{\displaystyle {\partial E^{(0)}} \bigl ( \textbf{x}^{\text {prec}(l, i, e)} \bigr )}{\partial x_i^{(e)}} \\ \qquad \qquad \qquad \qquad \qquad \qquad \text {if } i = j \text { and } q = e, \end{array} \right. \end{aligned}$$

(C29)

where

$$\begin{aligned} \textbf{x}^{\text {prec}(l, i, e)}=\left\{ \begin{array}{ll} \textbf{x}^{(l,i,e-1)} &{} \text { if } e \ne r; \\ \\ \textbf{x}^{(l,i-1,b)} &{} \text { if } i \ne 1, e = r; \\ \\ \textbf{x}^{(l-1,nm,b)} &{} \text { if } i = 1, e = r. \end{array} \right. \end{aligned}$$

(C30)

Algorithm 3

NL-SOR algorithm

Algorithm 3 allows to denote the image vectors actually defined at each iterate l, at every pixel i and at each color e. We observe that the algorithm here proposed is a particular case of that given in Subsection 4.1, and has been suitably modified in order to give a rigorous definition of the image vector ${\textbf{x}}$ at every iterate l, at every pixel i and at each color e.

Moreover, fixed the step (l, i, e), let ${\textbf{x}^{(l, i, e)} {\setminus } (x^{(l,i,e)})}_i^{(e)} \in {\mathbb {R}}^{3nm-1}$ be the vector whose elements are those of ${\textbf{x}}^{(l, i, e)}$ except ${(x^{(l,i,e))})}_i^{(e)}$. The value of this pixel ${(x^{(l,i,e))})}_i^{(e)}$ is an unknown variable, which we call z. For each fixed value of ${\textbf{x}}^{(l, i, e)} {\setminus } {(x^{(l,i,e))})}_i^{(e)}$, let us define the following energy function ${\overline{E}}^{(l,i,e)}:{\mathbb {R}} \rightarrow {\mathbb {R}}$ by

$$\begin{aligned} {\overline{E}}^{(l,i,e)} (z)= E^{(0)}({{\textbf{x}}}^{(l, i, e)} \setminus {(x^{(l,i,e))})}_i^{(e)}, z). \end{aligned}$$

(C31)

Theorem 3

Let $E^{(0)}:{\mathbb {R}}^{3nm} \rightarrow {\mathbb {R}}$ be a function of class $C^1$ and coercive, that is

$$\begin{aligned} \lim _{\Vert x\Vert \rightarrow +\infty } E^{(0)}(x)= +\infty \,; \end{aligned}$$

(C32)

fix ${{\textbf{x}}}^{(0, nm,b)} \in {\mathbb {R}}^{3nm}$, and let $\{ {{\textbf{x}}}^{(l,i,e)}\}$, $l \in {\mathbb {N}}, i=1,2,\ldots , 3\,m, e \in \{r,g,b\}$, be the sequence defined iteratively in (C29), where $0< \omega < 2$ and

$$\begin{aligned} \displaystyle { T > \max _{i=1,2,\ldots , nm, e = r,g,b} \, \max _\textbf{x} \, \left\{ \dfrac{\partial ^2_+ E^{(0)} (\textbf{x})}{(\partial x_i^{(e)})^2}, \dfrac{\partial ^2_- E^{(0)} (\textbf{x})}{(\partial x_i^{(e)})^2} \right\} }.\nonumber \\ \end{aligned}$$

(C33)

Let ${\overline{E}}^{(l,i,e)}:{\mathbb {R}} \rightarrow {\mathbb {R}}$ be the function defined in (C31). Assume that ${\overline{E}}^{(l,i,e)}$ is convex, admits both left and right derivative on ${\mathbb {R}}$ and is not secondly differentiable (at most) at a finite number of points.

Then, $\displaystyle { \lim _{(l,i,e)} \nabla E^{(0)} (x^{(l,i,e)})=0}$.

Proof

We begin by proving that, during the updating of ${(x^{(l,i,e))})}_i^{(e)}$, the function $E^{(0)}$ is non-increasing.

If

$$\begin{aligned} {{\overline{E}}^{(l,i,e)}}^{\prime } ({(x^{\text {prec}(l,i,e))})}_i^{(e)})= 0, \end{aligned}$$

(C34)

then, since in (C29) it is $\displaystyle {{(x^{\text {prec}(l,i,e))})}_i^{(e)}= \dfrac{\displaystyle {\partial E^{(0)}} \bigl (x^{\text {prec}(l, i, e)} \bigr )}{\partial x_i^{(e)}} }$, we get

$$\begin{aligned} (\textbf{x}^{(l, i, e)})_i^{(e)}= (x^{\text {prec}(l, i, e)})_i^{(e)} \end{aligned}$$

(C35)

and hence $\textbf{x}^{(l,i,e)}=\textbf{x}^{\text {prec}(l,i,e)}$, Moreover, the value of the energy function does not change.

Now, we treat the case when

$$\begin{aligned} {(x^{\text {prec}(l,i,e))})}_i^{(e)}\ne 0. \end{aligned}$$

(C36)

Note that, by (C32), we get

$$\begin{aligned} \lim _{z \rightarrow +\infty } {\overline{E}}^{(l,i,e)}(z)= \lim _{z \rightarrow -\infty } {\overline{E}}^{(l,i,e)}(z)= +\infty , \end{aligned}$$

(C37)

that is the function ${\overline{E}}^{(l,i,e)}$ is coercive on ${\mathbb {R}}$. Since ${\overline{E}}^{(l,i,e)}$ is also continuous, then, by [5, Theorem 2.32], ${\overline{E}}^{(l,i,e)}$ assumes the minimum value, say ${(t_*)}^{(l,i,e)}$.

We get that, for any $t > {(t_*)}^{(l,i,e)}$, the level set $L_t=\{z \in {\mathbb {R}}: {\overline{E}}^{(l,i,e)}(z) =t \}$ has exactly two points, and ${\overline{E}}^{(l,i,e)}(z)<t$ whenever z is in the interior of the interval whose endpoints are the elements of $L_t$.

Now we claim that, for every $t > {(t_*)}^{(l,i,e)}$, the level set $L_t=\{z \in {\mathbb {R}}: {\overline{E}}^{(l,i,e)}(z) = t^{(l,i,e)} \}$ has exactly two points.

Since ${\overline{E}}^{(l,i,e)}$ is convex and differentiable, we get that ${\overline{E}}^{(l,i,e)}(z)= {(t_*)}^{(l,i,e)}$ if and only if ${{\overline{E}}^{(l,i,e)}}^{\prime }(z)=0$. From the continuity of ${\overline{E}}^{(l,i,e)}$ and (C32) it follows that ${\overline{E}}^{(l,i,e)}$ assumes all values $t \in [{(t_*)}^{(l,i,e)}, + \infty [$. Since ${{\overline{E}}^{(l,i,e)}}^{\prime }$ is non-decreasing, then ${{\overline{E}}^{(l,i,e)}}^{\prime }$ is positive (resp. negative), and hence ${\overline{E}}^{(l,i,e)}$ is strictly increasing (resp. decreasing) at all greater (resp. smaller) points than the minimum points of ${\overline{E}}^{(l,i,e)}$. Thus, ${\overline{E}}^{(l,i,e)}$ assumes each value $t > {(t_*)}^{(l,i,e)}$ exactly two times, getting the claim.

Set $t^{(l,i,e)}= {\overline{E}}^{(l,i,e)}({(x^{\text {prec}(l,i,e))})}_i^{(e)})$, and $L_{t^{(l,i,e)}}= \{ {(x^{\text {prec}(l,i,e))})}_i^{(e)}, {\overline{z}}^{(l,i,e)}\}$, where ${\overline{E}}^{(l,i,e)}({\overline{z}}^{(l,i,e)})= t^{(l,i,e)}$.

Without loss of generality, let us consider the case ${{\overline{z}}}^{(l,i,e)}< {(x^{\text {prec}(l,i,e))})}_i^{(e)}$. Note that, in this case, ${{\overline{E}}^{(l,i,e)}}^{\prime } ({{\overline{z}}}^{(l,i,e)}) < 0$, while ${{\overline{E}}^{(l,i,e)}}^{\prime }({(x^{\text {prec}(l,i,e))})}_i^{(e)}) >0$.

By hypothesis, taking into account that ${\overline{E}}^{(l,i,e)}$ is of class $C^1$, from Lemma 2 applied to the interval $[{{\overline{z}}}^{(l,i,e)}, {(x^{\text {prec}(l,i,e))})}_i^{(e)}]$ we find $\xi \in ]{{\overline{z}}}^{(l,i,e)}, {(x^{\text {prec}(l,i,e))})}_i^{(e)}[$ and $\mu \ge 0$, such that

$$\begin{aligned}{} & {} \min \{ {{\overline{E}}^{(l,i,e)}}^{\prime \prime }_{-}(\xi ), {{\overline{E}}^{(l,i,e)}}^{\prime \prime }_{+}(\xi )\}\nonumber \\{} & {} \quad \le \mu \le \max \{ {{\overline{E}}^{(l,i,e)}}^{\prime \prime }_{-}(\xi ), {{\overline{E}}^{(l,i,e)}}^{\prime \prime }_{+}(\xi )\} \end{aligned}$$

(C38)

and

$$\begin{aligned} {\overline{E}}^{(l,i,e)} ({{\overline{z}}}^{(l,i,e)})= & {} {\overline{E}}^{(l,i,e)}({(x^{\text {prec}(l,i,e))})}_i^{(e)}) + \\ {}{} & {} + {{\overline{E}}^{(l,i,e)}}^{\prime } ({(x^{\text {prec}(l,i,e))})}_i^{(e)}) \\{} & {} \times ({{\overline{z}}}^{(l,i,e)} - {(x^{\text {prec}(l,i,e))})}_i^{(e)}) \\{} & {} + \dfrac{1}{2} \, \mu \, ({{\overline{z}}}^{(l,i,e)} - {(x^{\text {prec}(l,i,e))})}_i^{(e)})^2. \end{aligned}$$

Since ${\overline{E}}^{(l,i,e)}({(x^{\text {prec}(l,i,e))})}_i^{(e)})={\overline{E}}^{(l,i,e)} ({{\overline{z}}}^{(l,i,e)})$, then we have

$$\begin{aligned}{} & {} {{\overline{E}}^{(l,i,e)}}^{\prime } ({(x^{\text {prec}(l,i,e))})}_i^{(e)}) \, ( {{\overline{z}}}^{(l,i,e)} - {(x^{\text {prec}(l,i,e))})}_i^{(e)})+\nonumber \\{} & {} \quad + \frac{1}{2} \, \mu \, ({{\overline{z}}}^{(l,i,e)} - {(x^{\text {prec}(l,i,e))})}_i^{(e)})^2=0. \end{aligned}$$

(C39)

Note that from (C38) and (C33) we get

$$\begin{aligned} \mu \le T. \end{aligned}$$

(C40)

Now we claim that $\mu >0$. Indeed, if $\mu =0$, then from (C39) we get

$$\begin{aligned} {{\overline{E}}^{(l,i,e)}}^{\prime } ({(x^{\text {prec}(l,i,e))})}_i^{(e)}) \, ({{\overline{z}}}^{(l,i,e)} - {(x^{\text {prec}(l,i,e))})}_i^{(e)}) =0, \end{aligned}$$

and hence ${{\overline{z}}}^{(l,i,e)} = {(x^{\text {prec}(l,i,e))})}_i^{(e)}$, because ${{\overline{E}}^{(l,i,e)}}^{\prime } ({(x^{\text {prec}(l,i,e))})}_i^{(e)})>0$. This is absurd because ${{\overline{z}}}^{(l,i,e)} < {(x^{\text {prec}(l,i,e))}}_i^{(e)}$. Therefore, we get the claim. From (C39), we obtain

$$\begin{aligned} {(x^{\text {prec}(l,i,e))})}_i^{(e)} - {{\overline{z}}}^{(l,i,e)} =\dfrac{2}{\mu } \, {{\overline{E}}^{(l,i,e)}}^{\prime }({(x^{\text {prec}(l,i,e))})}_i^{(e)}).\nonumber \\ \end{aligned}$$

(C41)

We recall that, by (C29), it is

$$\begin{aligned} \displaystyle { {(x^{(l,i,e))})}_i^{(e)} = {(x^{\text {prec}(l,i,e))})}_i^{(e)} - \dfrac{\omega }{T} \, {{\overline{E}}^{(l,i,e)}}^{\prime } ({(x^{\text {prec}(l,i,e))})}_i^{(e)})}. \end{aligned}$$

(C42)

Since $0< \omega < 2$, from (C40), (C41) and (C42) we have

$$\begin{aligned} 0&< {(x^{(l,i,e))})}_i^{(e)} - {x^{\text {prec}(l,i,e)})}_i^{(e)}= {(x^{(l,i,e))})}_i^{(e)} = \nonumber \\&= \dfrac{\omega }{T} {{\overline{E}}^{(l,i,e)}}^{\prime }({(x^{\text {prec}(l,i,e))})}_i^{(e)}) < \dfrac{2}{T} {{\overline{E}}^{(l,i,e)}}^{\prime } {(x^{\text {prec}(l,i,e))}}_i^{(e)}) \le \nonumber \\&\le \dfrac{2}{\mu } {{\overline{E}}^{(l,i,e)}}^{\prime }{(x^{\text {prec}(l,i,e))})}_i^{(e)} = {(x^{\text {prec}(l,i,e))})}_i^{(e)} -{{\overline{z}}}^{(l,i,e)}. \end{aligned}$$

(C43)

From (C43) it follows that

$$\begin{aligned}{} & {} {{\overline{z}}}^{(l,i,e)}< {(x^{(l,i,e))})}_i^{(e)}< {(x^{\text {prec}(l,i,e))})}_i^{(e)} \quad \text {when } {{\overline{z}}}^{(l,i,e)} \nonumber \\{} & {} \quad < {(x^{\text {prec}(l,i,e))})}_i^{(e)}. \end{aligned}$$

(C44)

Analogously, it is possible to prove that

$$\begin{aligned}{} & {} {{\overline{z}}}^{(l,i,e)}> {(x^{(l,i,e))})}_i^{(e)}> {(x^{\text {prec}(l,i,e))})}_i^{(e)} \quad \text {when } {{\overline{z}}}^{(l,i,e)} \nonumber \\{} & {} \quad > {(x^{\text {prec}(l,i,e))})}_i^{(e)}. \end{aligned}$$

(C45)

Thus, in the case (C36), we get

${\overline{E}}^{(l,i,e)}({(x^{(l,i,e))})}_i^{(e)}) < {\overline{E}}^{(l,i,e)}({(x^{\text {prec}(l,i,e))})}_i^{(e)})$. Therefore, in both cases (C34) and (C36), the sequence $\{{\overline{E}}^{(l,i,e)}({(x^{(l,i,e))})}_i^{(e)})\}$, $l \in {\mathbb {N}}$, $i=1,2, \ldots , nm$, $e \in \{r,g,b\}$, is non-increasing. Since $E^{(0)}$ is bounded from below, then the sequence $\{{\overline{E}}^{(l,i,e)}({(x^{(l,i,e))})}_i^{(e)})\}$, $l \in {\mathbb {N}}$, $i=1,2, \ldots , nm$, $e \in \{r,g,b\}$, is non-increasing, and hence it is convergent.

Now, we claim that the sequence $\{{(x^{(l,i,e))})}_i^{(e)}-{(x^{\text {prec}(l,i,e))})}_i^{(e)} \}$, $l \in {\mathbb {N}}$, $i=1,2, \ldots , nm$, $e \in \{r,g,b\}$, converges to 0.

Fix $l \in {\mathbb {N}}$, $i=1,2, \ldots , nm$, $e \in \{r,g,b\}$. If ${{\overline{E}}^{(l,i,e)}}^{\prime } ({(x^{\text {prec}(l,i,e))})}_i^{(e)})= 0$, then, as seen in (C35), we get ${(x^{(l,i,e))})}_i^{(e)}={(x^{\text {prec}(l,i,e))})}_i^{(e)}$.

Now, we consider the case when ${{\overline{E}}^{(l,i,e)}}^{\prime } ({(x^{\text {prec}(l,i,e))})}_i^{(e)}) \ne 0$. By an argument similar to that used in the proof of [16, Theorem 2], from Lemma 2 applied to the interval whose endpoints are ${(x^{(l,i,e))})}_i^{(e)}$ and ${(x^{\text {prec}(l,i,e))})}_i^{(e)}$, we find a non-negative real number $\mu$ with

$$\begin{aligned}&{\overline{E}}^{(l,i,e)}({(x^{\text {prec}(l,i,e))})}_i^{(e)})- {\overline{E}}^{(l,i,e)}({(x^{(l,i,e))})}_i^{(e)}) \nonumber \\ {}&\quad = {{\overline{E}}^{(l,i,e)}}^{\prime } ({(x^{\text {prec}(l,i,e))})}_i^{(e)}) ({(x^{\text {prec}(l,i,e))})}_i^{(e)} - {(x^{(l,i,e))})}_i^{(e)}) \nonumber \\&\qquad - \dfrac{\mu }{2} \, ({(x^{(l,i,e))})}_i^{(e)} - {(x^{\text {prec}(l,i,e))})}_i^{(e)})^2. \end{aligned}$$

(C46)

From (C42) we get

$$\begin{aligned}&{(x^{(l,i,e))})}_i^{(e)} - {(x^{\text {prec}(l,i,e))})}_i^{(e)} = \nonumber \\&\quad = ( {(x^{(l,i,e))})}_i^{(e)} - {(x^{\text {prec}(l,i,e))})}_i^{(e)})^2 \nonumber \\&\qquad \times \dfrac{T}{\omega \, \, {{\overline{E}}^{(l,i,e)}}^{\prime } ({(x^{\text {prec}(l,i,e))})}_i^{(e)})}. \end{aligned}$$

(C47)

From (C46) and (C47) we obtain

$$\begin{aligned} 0< & {} {\overline{E}}^{(l,i,e)}({(x^{\text {prec}(l,i,e))})}_i^{(e)})- {\overline{E}}^{(l,i,e)}({(x^{(l,i,e))})}_i^{(e)}) = \nonumber \\= & {} \dfrac{T}{\omega } \, ( {(x^{(l,i,e))})}_i^{(e)} - {(x^{\text {prec}(l,i,e))})}_i^{(e)})^2 \nonumber \\{} & {} - \dfrac{\mu }{2} ( {(x^{(l,i,e))})}_i^{(e)} - {(x^{\text {prec}(l,i,e))})}_i^{(e)})^2 = \nonumber \\= & {} \left( \dfrac{T}{\omega } - \dfrac{\mu }{2} \right) \, ( {(x^{(l,i,e))})}_i^{(e)} - {(x^{\text {prec}(l,i,e))})}_i^{(e)})^2. \end{aligned}$$

(C48)

As $0< \omega < 2$ and $0 \le \mu < T$, we get

$$\begin{aligned} \displaystyle {\dfrac{T}{\omega }-\dfrac{\mu }{2} \ge T \, \left( \dfrac{1}{\omega } - \dfrac{1}{2} \right) >0}. \end{aligned}$$

(C49)

From (C48) and (C49) we obtain

$$\begin{aligned} 0\le & {} ( {(x^{(l,i,e))})}_i^{(e)} - {(x^{\text {prec}(l,i,e))})}_i^{(e)})^2 \le \nonumber \\ {}\le & {} \dfrac{2 \, \omega }{T\, (2 - \omega )} \, ( {\overline{E}}^{(l,i,e)}({(x^{\text {prec}(l,i,e))})}_i^{(e)})\nonumber \\{} & {} - {\overline{E}}^{(l,i,e)}({(x^{(l,i,e))})}_i^{(e)}) \,). \end{aligned}$$

(C50)

Note that (C50) holds also when ${{\overline{E}}^{(l,i,e)}}^{\prime } ({(x^{\text {prec}(l,i,e))})}_i^{(e)}) = 0$. Thus, in both cases (C34) and (C36), from (C50) and the convergence of the sequence $\{{\overline{E}}^{(l,i,e)}({(x^{(l,i,e))})}_i^{(e)})\}$, $l \in {\mathbb {N}}$, $i=1,2, \ldots , nm$, $e \in \{r,g,b\}$, it follows that the sequence $\{ ( {\overline{E}}^{(l,i,e)}({(x^{\text {prec}(l,i,e))})}_i^{(e)})- {\overline{E}}^{(l,i,e)}({(x^{(l,i,e))})}_i^{(e)}) \,) \}$, $l \in {\mathbb {N}}$, $i=1,2, \ldots , nm$, $e \in \{r,g,b\}$, converges to 0. From this, it follows that

$$\begin{aligned} \lim _{(l,i,e)} ( \, {(x^{(l,i,e))})}_i^{(e)}-{(x^{\text {prec}(l,i,e))})}_i^{(e)} ) =0, \end{aligned}$$

(C51)

getting the claim.

From (C29) we deduce

$$\begin{aligned} (x^{(l, i, e)})_j^{(q)}- (x^{\text {prec}(l, i, e)})_j^{(q)} = -\dfrac{\omega }{T} \, \dfrac{\displaystyle {\partial E^{(0)}} \bigl (\textbf{x}^{\text {prec}(l, i, e)} \bigr )}{\partial x_i^{(e)}}.\nonumber \\ \end{aligned}$$

(C52)

By arbitrariness of $i \in \{ 1,2, \ldots , nm\}$ and $e \in \{ r, g, b \}$, from (C51) and (C52) we deduce that $\displaystyle {\lim _{(l,i,e)} \nabla E^{(0)} (x^{(l,i,e)})}\displaystyle {=0}$, that is the assertion. $\square$

Appendix D: Componentwise Convexity of the First Approximation

Now, we prove that the first approximation is componentwise convex.

Theorem 4

When $p=0$, the function $E^{(p)}$ in (15) is componentwise convex.

Proof

We recall that $\overline{\varphi }$ is componentwise convex on ${\mathbb {R}}^2$ with respect to $t_1$ and $t_2$. Fix $k \in \{1,2,3\}$ and $c \in C_k$, and choose $\Xi ^k_c \in \{N^k_c,V^k_c\}$.

Now we claim that the function $\textbf{x} \mapsto \overline{\varphi }(\Xi _c^k \textbf{x}, \Xi _{\pi _k(c)} \textbf{x})$ is componentwise convex concerning the components of $\textbf{x}\in {\mathbb {R}}^{3nm}$. Indeed, fix $x_{i,j}^{(e)}$ with $i \in \{1,2,\ldots , n\}$, $j \in \{1,2,\ldots , m\}$ and $e \in \{r,g,b \}$.

Now we prove the convexity of $\overline{\varphi }$ with respect to the variable $x_{i,j}^ {(e)}$, in the following three cases:

I):

$(i,j ) \not \in c \cup \pi _k(c)$;

II):

$(i,j ) \in c$;

III):

$(i,j ) \in \pi _k(c)$.

We observe that it is impossible that $(i,j) \in c \cap \pi _k(c)$, thanks to our definition of $\pi _k(c)$.

Fix arbitrarily $u,w \in {\mathbb {R}}$ and $t \in [0,1]$.

In case I), note that $\Xi _c^k \textbf{x}$ and $\Xi ^k_{\pi _k(c)}\textbf{x}$ are independent of the value of the variable $x_{i,j}^{(e)}$. So, we get the function $\overline{\varphi }$ evaluated in $\Xi _c^k$, where all pixels are fixed except $x_{i,j}^{(e)}$.

Fixed an image $\textbf{x}$, let $\textbf{x} {\setminus } x_{i,j}^{(e)} \in {\mathbb {R}}^{3nm-1}$ be the vector whose elements are those of ${\textbf{x}}$ with the exception of $x_{i,j}^{(e)}$. Observe that the value of this pixel $x_{i,j}^{(e)}$ is an unknown variable. We have:

$$\begin{aligned}{} & {} \overline{\varphi }(\Xi _c^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}\\{} & {} \quad = tu+(1-t)w), \Xi _{\pi _k(c)}^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)} = tu+(1-t)w) )=\\{} & {} \quad = t \, \overline{\varphi } (\Xi _c^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}= u), \Xi _{\pi _k(c)}^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}= u)) \\{} & {} \qquad + (1-t) \overline{\varphi } (\Xi _c^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}= w), \\{} & {} \qquad \times \Xi _{\pi _k(c)}^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}= w)), \end{aligned}$$

since

$$\begin{aligned}{} & {} \Xi _c^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}=a)= \Xi _c^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}=b) \quad \text {and \, }\nonumber \\ \end{aligned}$$

(D53)

$$\begin{aligned}{} & {} \Xi _{\pi _k(c)}^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}=a)= \Xi _{\pi _k(c)}^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}=b)\nonumber \\ \end{aligned}$$

(D54)

for each $a, b \in {\mathbb {R}}$.

Now we deal with the case II).

It is not difficult to see that since the finite difference operators $D_c^k$ are linear and the norm $\Vert \cdot \Vert _2$ is a convex function, the operators $\Xi _c^k$ and $\Xi _{\pi _k(c)}$ are convex on their domain. We get

$$\begin{aligned}&\overline{\varphi }(\Xi _c^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}\nonumber \\&\quad = tu+(1-t)w), \Xi _{\pi _k(c)}^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)} = tu+(1-t)w) ) \le \nonumber \\&\quad \le \overline{\varphi }(t \, \Xi _c^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}= u) + (1-t)\Xi _c^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}= w), \nonumber \\&\qquad \times \Xi _{\pi _k(c)}^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}= \nonumber \\&\quad = tu+(1-t)w) )\le \nonumber \\&\quad \le t \, \overline{\varphi } (\Xi _c^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}= u), \Xi _{\pi _k(c)}^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}= u)) \nonumber \\&\qquad + (1-t)\overline{\varphi } (\Xi _c^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}= w), \nonumber \\&\qquad \times \Xi _{\pi _k(c)}^k (\textbf{x} \setminus x_{i,j}^{(e)}, x_{i,j}^{(e)}= w)). \end{aligned}$$

(D55)

The first inequality in (D55) holds, since $\Xi _c^k$ is (globally) convex. Note that $\overline{\varphi }$ is increasing in the first component.

Furthermore, the third inequality in (D55) follows from (D54), since the function $(t_1,t_2) \mapsto \overline{\varphi } (t_1,t_2)$ is componentwise convex with respect to the variables $t_1$ and $t_2$, and since $(i,j) \not \in \pi _k(c)$.

Case III) is analogous to the case II). Thus, the assertion follows.

$\square$