Abstract
This paper proposes an edgepreserving regularization technique to solve the color image demosaicing problem in the realistic case of noisy data. We enforce intrachannel local smoothness of the intensity (lowfrequency components) and interchannel local similarity of the depth of object borders and textures (highfrequency components). Discontinuities of both the lowfrequency and highfrequency components are accounted for implicitly, i.e., through suitable functions of the proper derivatives. For the treatment of even the finest image details, derivatives of first, second, and third orders are considered. The solution to the demosaicing problem is defined as the minimizer of an energy function, accounting for all these constraints plus a data fidelity term. This nonconvex energy is minimized via an iterative deterministic algorithm, applied to a family of approximating functions, each implicitly referring to geometrically consistent image edges. Our method is general because it does not refer to any specific color filter array. However, to allow quantitative comparisons with other published results, we tested it in the case of the Bayer CFA and on the Kodak 24image dataset, the McMaster (IMAX) 18image dataset, the Microsoft Demosaicing Canon 57image dataset, and the Microsoft Demosaicing Panasonic 500image dataset. The comparisons with some of the most recent demosaicing algorithms show the good performance of our method in both the noiseless and noisy cases.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
The demosaicing problem arises from acquiring RGB color images through CCD (charged coupled devices) digital cameras. In the RGB model, each pixel of a digital color image is associated with a triplet of numbers, representing the intensity of the red, green, and blue, respectively. However, most commercial cameras employ a single sensor associated with a color filter that only permits, at each pixel, the measurement of the reflectance of the scene at one of the three colors, according to a predefined scheme or pattern, called color filter array (CFA), as illustrated in Fig. 1. This situation implies that, for each pixel, the other two missing colors must be estimated. Although several diverse CFAs have been proposed for the acquisition (e.g., see [31] and [35] for the square pixel layout and [3] for the Penrose aperiodic pixel layout), the most largely diffuse is the Bayer pattern [4], shown in Fig. 2. Most of the literature on demosaicing is thus devoted to algorithms designed with explicit reference to the Bayer pattern. In [30, 46], and [56], comprehensive surveys of the state of the art can be found. In the Bayer scheme, green is sampled in a number of pixels double the number where red and blue are measured to exploit the human eye’s higher sensibility to the green wavelength.
By decomposing the recorded digital image into the three color channels, one obtains three downsampled grayscale images so that demosaicing could be interpreted as the problem of interpolating grayscale images from sparse data.
In most recreational cameras, demosaicing is part of the processing pipeline required to convert these images into a viewable format. The camera’s builtin firmware is substantially based on very fast, or even realtime, interpolation algorithms.
Nevertheless, interpolation is not sufficient to produce the highquality images required, e.g., for professional purposes. Indeed, interpolation is often applied to each channel separately, disregarding the mutual solid correlation between the channels. Since interpolation algorithms are basically lowpass filters, they perform well in the lowfrequency components but reduce the highfrequency ones, thus producing color artifacts in the output image. A remedy to this problem should consider the high correlation among the three color channels, which occurs mainly in correspondence with their highfrequency components. Indeed, the image edges, such as object borders and textures, are largely shared by the color channels. Unavoidably, exploiting the interchannel information requires more complex algorithms, which are usually timeconsuming and cannot run onboard.
Nowadays, many professional digital cameras can save images in a raw format, thus allowing users to demosaic them using offline software. When the raw image data is accessible, one can use various demosaicing algorithms instead of being limited to the one built into the camera. For example, the raw development program RawTherapee [64] gives the user an option to choose, among around ten different algorithms, which should be used. For the Bayer CFA, AMaZE (Aliasing Minimization and Zipper Elimination), developed by Emil J. Martinec in 2010, yields the best results in most cases. Nevertheless, since different demosaicing algorithms present differences in rendering the finest detail and grain texture, photographers often prefer a particular algorithm for aesthetic reasons related to this effect. On the other hand, when it comes to professional applications, computational time is less critical. Thus, there is an interest in developing dedicated and even very sophisticated algorithms that can provide demosaiced images where the highfrequency components are accurately reconstructed.
To this end, this paper proposes an algorithm for image demosaicing that falls within the framework of the edgepreserving regularization approaches and is suited, naturally, to deal with noisy data. More precisely, we propose an algorithm for joint demosaicing and denoising. Regularization requires the adoption of constraints for the solution. The constraints we consider here are intrachannel (spatial) and interchannel (spectral) local correlation. Concerning the intrachannel correlation, we assume the intensity of each channel to be locally regular, i.e., piecewise smooth, so that noise can also be removed. We describe this constraint through stabilizers that discourage intensity discontinuities of first, second, and third order in a selective way, i.e., permitting to discontinuities associated with actual edges to emerge. This allows us to describe scenes that are even very complex. Indeed, firstorder local smoothness characterizes images consisting of constant patches, secondorder local smoothness describes patches whose pixels have values varying linearly, while thirdorder local smoothness is used to represent images made up of quadraticvalued patches. As per the interchannel correlation, we enforce it in correspondence with the intensity discontinuities using constraints that promote their amplitude in the three channels to be equal almost everywhere.
Note that all these constraints are not biased in favor of one of the three channels, nor is the geometry of the sampling pattern in any way exploited. Thus, the method we propose is entirely independent of the CFA considered, although, in the experimental result section, we present its application to images mosaiced through the Bayer CFA.
All the above constraints, including the data fidelity term, are merged in a nonconvex energy function, whose minimizer is our desired solution.
The nonconvexity characteristics of the energy function preclude its minimization through conventional gradient descent optimization techniques. Stochastic and deterministic methodologies offer alternative avenues for minimizing nonconvex energy functions. While stochastic techniques often yield exact outcomes, their computational demands are considerable (cf. [25]). Conversely, deterministic algorithms, while not guaranteeing convergence to the global optimum, facilitate the attainment of satisfactory reconstructions within reduced computational durations (cf. [6, 8]).
Here, we adopt the graduated nonconvexity (GNC) algorithm [9], which is a prevalent one within the deterministic approaches. This technique approximates the energy function through progressively refined approximation functions converging toward the original function. Each approximation function is optimized using conventional optimization algorithms, leveraging the minima attained in the preceding approximation as initial values.
As mentioned, our edgepreserving regularization approach, combined with the GNC minimization strategy, can produce image solutions that exhibit reliable and geometrically consistent discontinuities of both the intensity and the gradients despite the necessary smoothness constraints.
In the first works proposing edgepreserving regularization, the image discontinuities were often represented by extra, explicit variables, the socalled line processes [25]. That way, it was relatively easy to formulate their required properties regarding constraints. Nevertheless, the use of explicit line variables entails significant computational costs. Thus, socalled duality theorems were derived (see, e.g., [15, 26, 14]) to demonstrate the edgepreserving properties of suitable stabilizers without introducing extra variables. In particular, we developed duality theorems to determine the properties required for stabilizers to manage lines with the desired regularity features implicitly. In this paper, we choose a suitable family of approximations for the GNC having the peculiarity that each function satisfies the conditions required for an implicit treatment of geometrically significant edges, as expressed in the duality theorems proposed in [14]. This allows for better adherence of the approximations to the ideal energy function, resulting in better coherence with the properties required for the desired solution.
GNC, initially introduced in [9], has seen significant advancements over the years. In [7], the GNC framework was extended to accommodate the restoration of noisy images, by incorporating considerations on the geometry of discontinuities. In [60], blur in noisy images has been considered as well. Since then, [15] proposed GNC for deblurring and denoising tasks, enforcing constraints such as line continuation or line nonparallelism, and a GNC variant was designed to deal with point spread functions with a vast domain [12].
The GNC methodology has found applications across diverse domains, which stand as a testament to its versatility and efficacy. Indeed, it has been successfully utilized in addressing the combinatorial data analysis challenge of seriation, as evidenced in [23]. Moreover, its utility extends to tackling stochastic problems, as discussed in [32], and resolving combinatorial optimization dilemmas defined on the domain of partial permutation matrices, as explored in [48]. Additionally, the GNC framework has been harnessed for addressing the maximum a posteriori inference problem, elucidated in [49], as well as for tasks such as pose estimation, documented in [67], and spatial perception, outlined in [71].
This paper extends the GNC methodology by explicitly focusing on the nonparallelism constraint for the discontinuities and proposing a first, initial approximation that is componentwise convex. We call this new version of GNC graduated componentwise nonconvexity (GCNC).
The paper is organized as follows. In Sect. 2, stateoftheart data in the color image demosaicing and denoising field are surveyed. Section 3 is devoted to formulating the problem and adopting the specific edgepreserving regularization strategy. In Sect. 4, the solution algorithm is described in detail. Section 5 is devoted to the quantitative comparison between the results obtained with our method and those of some of the most performing algorithms proposed in the recent literature, using the Kodak image dataset [43], the McMaster image dataset [75], the Microsoft Demosaicing Canon Dataset [38], and the Microsoft Demosaicing Panasonic Dataset [38] as benchmark sets, and with specific reference to the Bayer CFA. Conclusions and suggestions for prospects are given in Sect. 6, and finally, in the Appendices, some mathematical aspects are developed in detail.
2 Related Work
A significant problem of demosaicing is avoiding oversmoothing of the edges, so the fundamental feature of any method is its ability to perform interpolation along and not across the edges. Some methods perform directional interpolation by analyzing the variance of the color differences to exploit the high correlation between the color planes [18]. For example, the work in [2] proposes highorder interpolation and Sobel operators to compute the gradients, and in [24], a level set method is used to minimize an energy function that gives the direction of the edges. In [19], the interpolation direction is chosen by exploiting an edgesensing parameter called integrated gradient, which simultaneously considers color intensity and color difference. In general, due to solid correlations existing among three color channels in nature, performing interpolation in the relatively smooth color difference field can be easier and more convenient.
In other methods, the best reconstruction of the missing data, first estimated by interpolating along horizontal and vertical directions, is chosen [33, 52], or the two reconstructions are fused [74, 75]. In particular, [33] proposes an algorithm based on the Laplacian filter by selecting the interpolation directions having the most minor level of color artifacts. Instead, the method in [17] infers the missing colors by considering the local image geometry through the image selfsimilarity.
As an alternative to color difference interpolation, the algorithms in [41] and [72], among others, are based on interpolation in a residual domain, where the residuals are differences between observed and tentatively estimated pixel values. This approach is justified because the residual field is smoother than the color difference data field. Hence, the methods based on residual interpolation may be advantageous regarding both peak signaltonoise (PSNR) and subjective visual quality. In [59], the adaptive residual interpolation method is proposed to combine the residual interpolation [39] and the minimizedLaplacian residual interpolation [40]. The developed method is extensively tested on large datasets in [36].
In [29] and [45], the strong correlation between the high frequencies of the three color components is directly exploited. In particular, the algorithm presented in [29] forces similarity between the high frequencies of the red and the green and of the blue and the green.
The sparse nature of color images has also been exploited for demosaicing. A suitable dictionary is designed and applied with the iterative KSVD algorithm in [50], whereas in [58], the dictionary is constructed based on a clear distinction between the interchannel and intrachannel correlations of natural images and the sparse representation of the image is found through compressed sensing. In [1], a locally adaptive approach is used for demosaicing dualtree complex wavelet coefficients.
Within regularization approaches, the totalvariation principle is used in [66], while [54] proposes first a general quadratic smoothness regularizer and then an adaptive filter in order to improve the reconstruction near the edges of the first estimate.
The methods surveyed above have been mainly designed for noisefree data. An abundance of literature has also been devoted to the more realistic noisy data scenario. In this context, performing denoising as a preor postprocessing has significant drawbacks. In the first case, denoising must be separately performed on the individual channel so that the entire image resolution cannot be exploited. In addition, denoising alters the color samples in the raw image. On the other hand, if demosaicing is performed first, the interpolation process changes the noise distribution. For instance, in [55] [54], the authors evaluate the statistical characteristics of the noise resulting from the demosaicing process performed through spacevarying filters, and then design an ad hoc postprocessing denoising strategy.
The method in [76] works in two steps. First, the fullresolution green component is recovered from the difference signals of the color channels by exploiting both spectral and spatial correlations to suppress sensor noise and interpolation error simultaneously. Second, the CFA channeldependent noise is removed from the reconstructed green channel with a waveletbased approach, and the red and blue channels are also estimated and denoised. The work in [34] presents an algorithm that uses a modified total least squared estimation technique to estimate an ideal demosaicing filter to deal with the noise affecting the data.
In [20], a noisy mosaiced image’s luminance and chrominance channels are first reconstructed by exploiting a frequency analysis of the sampling pattern induced by the Bayer CFA. Wiener filters are then designed to denoise the chrominances, whereas the luminance is linearly filtered as a grayscale image. An extended variant of this approach is also proposed, in which the demosaiced image is mosaiced again and then demosaiced using the method in [74].
Regularization is an ideal setting for joint demosaicing and denoising, which can be performed simultaneously based on the same image priors. These can be manually designed, as in [21], where a total variation (TV) prior ensures the smooth property of the image or can be learned from the image dataset, as in [38], where regression tree fields are used to learn realistic image datasets. The method in [42] models the problem as a minimization problem and uses an image dataset to improve performance.
In recent years, many demosaicing algorithms have been developed by exploiting the potential of convolutional neural networks (CNNs) to avoid dependence on empirically defined priors. For instance, in [27], a deep demosaicing and denoising network was trained on millions of artificially degraded images. A twostage CNN and a threestage CNN have been proposed in [69] and [22], respectively, and in [68], an image demosaiced using traditional interpolation schemes is refined using the residual network. The work in [44] a specific network architecture is designed, inspired by powerful classic image regularization and largescale optimization. To improve the quality of the recovered image, in [47], the guidance of the density map and the exploitation of edge characteristics are proposed, and in [70], the structure of the CNN model and the loss functions are carefully studied. Finally, in [63], a CNN model is used as a prior within a regularization setup.
Although deep learningbased methods have demonstrated higher performance, they are highly datadependent and require many training images, i.e., fullresolution RGB images. In many approaches, the authors have used as reference images already processed; that is, they apply a mosaic mask on images already demosaiced, thus obtaining unrealistic training pairs [68]. In fact, with this learning strategy, the main problem is that demosaicing the training data artifacts will hamper the reconstruction’s performance and overall quality. Furthermore, methods based on learning require high memory and computation costs, which is undesirable for integrated sensor systems. For these reasons, interpolationbased methods are still widely applied in practical use due to their simplicity.
3 Formulation of the Demosaicing Problem and Its Regularization
3.1 The Data Generation Model
A color image of size \(n\times m\) can be represented as a vector \(\textbf{x}\in {\mathbb {R}}^{3nm}\), \(\textbf{x}=\left( (\textbf{x}^{(r)})^T\, (\textbf{x}^{(g)})^T\,(\textbf{x}^{(b)})^T \right) ^T\), where \(\textbf{x}^{(r)}, \textbf{x}^{(g)}, \textbf{x}^{(b)} \in {\mathbb {R}}^{nm} \) are the red, green and blue channels expressed in the lexicographic notation, respectively. The mosaicing problem is formulated as
where \(\textbf{x}, \textbf{y}, \textbf{n} \in [0,255]^{3nm}\) denote the ideal color image, the mosaiced image, and the additive noise, respectively. We assume the noise to be independent, Gaussian, with null mean and variance \(\sigma ^2\). The matrix \(M\in \{0,1\}^{3nm\times 3nm }\) is a linear operator associated with the acquisition pattern, consisting of the following block diagonal matrix:
where \(O \in {\mathbb {R}}^{nm\times nm}\) is the null matrix, and \(M^{(r)}\), \(M^{(g)}\), \(M^{(b)}\) are diagonal matrices in \({\mathbb {R}}^{nm\times nm}\). For the Bayer pattern, the diagonal elements of these matrices are given by
where (i, j) is the generic pixel index.
The demosaicing problem is the inverse problem associated with the direct problem formulated in (1) and consists of finding an estimate \(\widetilde{\textbf{x}}\) of the ideal image, given the mosaiced image \(\textbf{y}\) and the operator M. Since M is singular, the demosaicing problem is illposed in the Hadamard sense because, in general, it does not admit a unique solution. Given \(\textbf{y}\), there are infinitely many feasible solutions since, at each pixel, the values of the two unmeasured channels do not contribute to the data. Therefore, regularization techniques are necessary to reduce the number of solutions.
3.2 The Regularization Model
We define our regularized solution \(\widetilde{\textbf{x}}\) as an argument of the minimum of the following energy function:
where \(\left\ \cdot \right\ _2 \) denotes the Euclidean norm, and the first term of the right hand of (4) expresses a data fidelity constraint, which is identically null in the noiseless case. The second term in the right hand of (4) regulates the intrachannel smoothness of the involved image. The third term imposes a correlation between the different channels, i.e., an interchannel smoothness. Intrachannel and interchannel smoothness are measured through the operators \(N_c^k\) and \(V_c^k\), respectively, and \(\varphi \) is a stabilizer that weights the degree of smoothness required and relaxes it when a discontinuity is expected.
Let us start by analyzing the form of the operator \(N_c^k\), given by
where \(D^k_c\) is a finite difference operator of order k applied to a suitable set c of adjacent pixels, called clique of order k. Therefore, from (5), it appears that \(N_c^k\) is the norm of the vector of the finite differences of the intensities of the red, green, and blue channels computed on the clique c of order k. The set \(C_k\) collects all cliques of order k. Each of such cliques is uniquely associated with a discontinuity of order k, labeled by a hidden line element.
To reconstruct the finest details in the images, we consider finite differences and then discontinuities of the first, second, and third order, that is \(k=1,2,3\). The geometry of the associated cliques is described in Appendix A.
The edges of the first order separate homogeneous patches in the image, the edges of the second order mark the slope of linearly varying areas, and the edges of the third order are associated with the intensity discontinuities in regions where intensity varies quadratically.
As the interchannel correlation aims to maintain the clue of the objects in the image, the finite difference operators should behave similarly in all three channels. So we define the operator \(V_c^k\) as follows:
which is the norm of the vector of the interchannel differences of the intrachannel korder derivatives. Again, a hidden line variable is implicitly associated with the clique c for each order \(k=1,2,3\). These further sets of hidden line variables mark the discontinuities between areas having homogeneous clues.
In (4), \(N_c^k\) and \(V_c^k\) are weighted by suitable stabilizers. These stabilizers should regulate the degree of smoothness required in the two cases and relax it when discontinuities are expected and dependent on their amplitude. In (4), we adopted the same parametric stabilizer \(\varphi \) for both the operators \(N_c^k\) and \(V_c^k\) and let its parameters possibly vary in the two terms (see also [14]).
For a more accurate reconstruction, edges must not be thick, or the object contours must not be blurred equivalently. To this aim, it is advisable to inhibit the creation of discontinuities at two adjacent cliques. Specifically, to prevent double edges of order k, simultaneous discontinuities at the cliques c and the previous one \(p_k(c)\) should be inhibited (see Appendix A for the definition of adjacent cliques).
When \(p_k(c)\) is not defined for the mixed cliques and the cliques on the border of the image, we automatically assume that the adjacent discontinuity is null.
Having in mind the abovedescribed properties to be featured by the stabilizer, we adopt herein a bivariate function \(\varphi : {\mathbb {R}} \times {\mathbb {R}} \rightarrow {\mathbb {R}}\) (see also [15]), defined by
where
and \(\zeta \) is chosen in such a way that \(\zeta  s\) is a positive and sufficiently small quantity, and for \(i=1, 2\) it is
\(\tau \) is a large enough real constant, and
The graph of the function in (7) for specific values of the parameters is shown in Fig. 3 (a). Note that when \(\zeta  s=0\) and \(\tau \) tends to \(+ \infty \), we have an ideal model with Boolean line variables (see Fig. 3 (b)). We have set \(\zeta =s+10^{4}\) and \(\tau =10^4\) in our experimental results.
In (7)–(12) \(\lambda ^2\) is a regularization parameter, which has the role of determining the smoothness of the solution, \(\alpha \) is a cost, which we have to pay whenever we introduce the image discontinuities. When there are some parallel discontinuities, we add the quantity \(\varepsilon \) to this cost. Thus, the value s in (8) is just the threshold necessary to introduce a discontinuity in the image when there are no parallel discontinuities.
In general, the stabilizer’s analytical form determines the amplitude of the discontinuities in the reconstructed image by promoting onoff discontinuities of large amplitude above a given threshold or more slowly varying discontinuities of graded amplitudes. In the primaldual formalism, the first type’s stabilizers implicitly address “hard”, Boolean line elements, while the second type addresses hidden “soft” line elements, ideally valued in [0, 1].
We recall that the functions \(g_i\), \(i=1,2\), defined in (9), are approximations of class \(C^1\) of the classical truncated parabola defined in [9] (see also [10, 14, 15]) that, when used as a stabilizer, implicitly addresses a Boolean line process. In the bivariate case, the function \(\varphi \) defined in (7) possesses the same characteristic when \(\tau \) tends to \(+ \infty \) and \(\zeta \) is very close to s. The actual form we propose is an approximation of such a function, with the property of class \(C^1\), which is essential for converging the minimization algorithm (see Subsection 4.1).
4 Graduated Componentwise NonConvexity Algorithm
We observe that the function \(\varphi \) defined in (7), as a function of two variables, is not convex, and hence neither is the energy function \(E(\textbf{x})\), defined in (4) as a function of \(3 \, n \, m \) variables. Thus, to minimize E, we determine a finite family of approximating functions \(\{E^{(p)}\}_p\), where \(E^{(0)}\) is componentwise convex, and \(E^{(2)}\) is the original energy function E. The initial point to minimize the componentwise convex approximation is found using the Local Edge Preserving (LEP) algorithm used in [13] for demosaicing. The LEP is a highspeed algorithm consisting of two phases. In the first phase, the missing components are determined by a weighted mean, which guarantees the preservation of the edges. In the second phase, the differences between the colors of the channels are imposed to be constant within homogeneous areas. Our algorithm in its whole is called graduated componentwise nonconvexity (GCNC) and is presented in Algorithm 1.
Note that our algorithm can be seen as a variant of the graduated nonconvexity (GNC) algorithm (see also [7, 15, 57, 60,61,62, 65]).
To construct the first componentwise convex approximation \(E^{(0)}\), we find a componentwise convex approximation for the stabilizers in (7), since the data term in (4) is globally convex. Such componentwise convex approximations can be constructed based on a componentwise convex approximation of the bivariate function \(\varphi \), as shown in Appendix D. To do this, we proceed as follows. First of all, we approximate the functions \(g_i(t_1)\) with the following convex approximations given by
(see also [60]). Moreover, we find an approximation, convex concerning the variable \(t_2\), of the function \(\varphi \) defined in (7), in the following way:
where \({\overline{t}}\) is the maximum value that a finite difference operator can assume (\({\overline{t}}=2^k \cdot \sqrt{2} \cdot 255\), \(k=1,2,3\), for the light intensity of the images in the range [0,255]). It is easy to check that \(g_2  g_1\) is convex since \(0< q_1 < q_2\).
We recall that \(t_1=N^k_c \textbf{x}\), \(t_2=N^k_{p_k(c)} \textbf{x}\) in the second term of the right hand of (4), and \(t_1=V^k_c \textbf{x}\), \(t_2=V^k_{p_k(c)} \textbf{x}\) in the third term of the right hand of (4). Let us fix \(k \in \{1,2,3\}\) and \(c \in C_k\), and choose \(\Xi ^k_c \in \{N^k_c,V^k_c\}\). So, \(t_1\) is a function of \(\textbf{x}\), and \(t_1(\textbf{x})= \Xi ^k_c(\textbf{x})\). In particular, \(t_1\) depends only on the variables involved in the clique c. Analogously, \(t_2(\textbf{x})= \Xi ^k_{p_k(c)}(\textbf{x})\) depends only on the variables involved in the clique \(p_k(c)\). Note that the function \(\overline{\varphi }\) defined in (14) is componentwise convex. However, the function \(\overline{\Phi }(\textbf{x})=\overline{\varphi } (t_1(\textbf{x}),t_2(\textbf{x}))\) is not componentwise convex concerning the components of the vector \(\textbf{x} \in {{\mathbb {R}}}^{3nm}\). This is because \(c \cap p_k(c) \ne \emptyset \). However, if we choose \(t_2(\textbf{x})= \Xi ^k_{\pi _k(c)}(\textbf{x})\) instead of \(\Xi ^k_{p_k(c)}(\textbf{x})\), then the function \(t_2\) is componentwise convex with respect to \(\textbf{x}\), as shown in Appendix D, since \(c \cap \pi _k(c)=\emptyset \). Thus, to define the family of the approximations for the algorithm GCNC, we proceed as follows.
If \(p \in [0,1]\), put
When \(p \in [1,2]\), set
where
In Appendix B, we will prove that for each \(p \in [1,2]\), \(\varphi ^{(p)}\) satisfies the duality conditions that guarantee the edgepreserving properties and the inhibition of double edges [14]. Note that, for \(p \in [0,1]\), the stabilizer \(\varphi (t_1, t_2)\) is equal to \(\overline{\varphi }(t_1, t_2)\), and hence fulfils the same properties. Furthermore, in [11], it is proved that the associated line process is nonBoolean. The hidden line elements become Boolean as far as p tends to 2. However, we experimentally observed that, in natural images, graded discontinuities can help prevent the aliasing effect. Thus, in the experiments, we stop the minimization algorithm at a suitable value of p different from 2, as explained in Sect. 5.
4.1 The NLSOR Algorithm
To minimize each approximation \(E^{(p)}\), we use a nonlinear successive over relaxation (NLSOR) algorithm, which is widely used in the literature (see also [7, 9, 16]). The NLSOR is defined as in Algorithm 2, where \(\epsilon >0\) is a fixed threshold, \(\omega >0\) is the accelerator parameter,
and the symbols \(\partial ^2_+\) and \(\partial ^2_\) denote the right and left second partial derivatives, respectively.
In our experimental results, we have set \(\omega =0.2\) and \(\epsilon =3 \sqrt{nm}\), where \(n\times m\) is the dimension of the involved image. We chose these values after a long experimental phase, where we monitored the trend of the energy function in each step of the algorithm to obtain highperformance results without excessive computational time.
In [16, Theorem 2], the convergence of the algorithm is proved when \(E^{(p)}\) is strictly convex and of class \(C^2\). However, such a theorem cannot be applied to our setting since our first approximation is componentwise convex and \(C^1\) but neither strictly convex nor \(C^2\). Thus, in Appendix C, we propose an extension of [16, Theorem 2] to prove that in our case, when \(p=0\), the algorithm stops in correspondence with a stationary point. Note that for \(p\ne 0\), we have no guarantee of converging to a stationary point of \(E^{(p)}\) using the NLSOR algorithm. However, the experimental results confirm that excellent solutions can still be achieved (see Sect. 5).
5 Experimental Results
In assessing the efficacy of the proposed demosaicing algorithm, we analyze diverse color image datasets, encompassing the Kodak image dataset [43], the McMaster image dataset [75], the Microsoft Demosaicing Canon Dataset [38], and the Microsoft Demosaicing Panasonic Dataset [38]. The Kodak dataset comprises 24 comprehensive fullcolor images sized at \(512 \times 768\) pixels each. Conversely, the McMaster image dataset features 18 images with dimensions of \(500 \times 500\) pixels, derived from original highresolution images sized at \(2310 \times 1814\) pixels. Renowned for their widespread adoption as benchmark datasets within demosaicing and various other color image processing domains, the Kodak and McMaster datasets serve as pivotal standards.
Moreover, the Microsoft Demosaicing Dataset Canon encompasses 57 images at \(210 \times 318\) pixels, while the Microsoft Demosaicing Dataset Panasonic comprises 500 images of identical dimensions. These raw images originate from distinct camera sources, namely a Canon EOS 550D and a Panasonic Lumix DMCLX3, respectively. Such datasets collectively facilitate a comprehensive evaluation of the proposed demosaicing algorithm across varied imaging scenarios and devices.
We implement the proposed algorithm in C language and run it in a Linux Ubuntu environment on a computer with an i59400F processor at 2.90 GHz. However, minimizing a nonconvex function is a complex task; therefore, high computational costs are still necessary. For example, in the case of the Kodak set, the proposed algorithm has an average computation time of 12.493 min for each image.
The free parameters \(\lambda \), \(\alpha \), and \(\varepsilon \) appearing in our energy function have been calibrated on the images of the Kodak dataset and then used for the other datasets. We employed a trialanderror strategy to look for the parameters that give the best average color peak signaltonoise ratio (CPSNR) on all the 24 images. The CPSNR quality index, a key metric in our calibration process, is defined as:
where MSE is the mean square error between the original image and the demosaiced one that, for color images, is defined as the arithmetic average of the mean square errors on the three channels.
We chose to increase p with a step of 0.01. Note that, for \(p=2\), the discontinuities present in the reconstructed image are Boolean, whereas, for \(p\ne 2\), we have reconstructions with nonBoolean line variables, as stated in the duality theorem given in [15].
Reconstructions obtained with Boolean lines are often disturbed by the aliasing effect, so that it may be preferable to stop the algorithm at a value \({\bar{p}}\) smaller than 2. The optimal value of \({\bar{p}}\) has been set as the one for which the sum of the MSEs of all the 24 Kodak images reconstructed with that p is minimum. Calling \(\eta _j(p)\) the MSE between the ideal image and the minimizer of the approximated energy function for the \(jth\) image at a given p, it is:
The proposed algorithm has been applied to both noiseless and noisy images, and the performance of the method has been compared with some of the most popular and bestperforming methods in the literature.
5.1 Noiseless Images
In the energy E (4), stabilizer \(\varphi \) depends on the free parameters \(\lambda \), \(\alpha \) and \(\varepsilon \). In the second addend of the right hand of (4), we used \(\varphi \) with the free parameters reported in Table 1, which we found to be the best for noiseless mosaiced images.
In the third term of the right hand of (4), the best free parameters for stabilizer \(\varphi \) to be used in the noiseless case are indicated in Table 2.
Given all these free parameters, we found \({{\bar{p}}} = 1.42\).
For the noiseless Kodak dataset, we compared the proposed method with the algorithms in [29, 33, 50, 1, 2, 19, 54] [58, 41], and [59]. In particular, we used the source code available on the web to obtain the results of the algorithm in [59]. The results obtained on the Kodak dataset are reported in Table 3. As usual with the Kodak images, we removed the 3 pixels wide external frame to compute the errors. Our method exhibits the highest CPSNR in more images than the other methods (the highest CPSNR is highlighted in boldface for each image). Furthermore, the proposed method achieves the best result on average.
For the noiseless McMaster dataset, a comparison has been made with the algorithms in [17, 29, 33, 41, 75], and [59], respectively. In particular, to collect the results of the algorithm in [29], we used the original MATLAB code provided by the authors, and for the results of the algorithms in [41] and [59], we used the source code available on the web. The results obtained are reported in Table 4. The proposed method achieved the best result in more than \(44\%\) of cases. However, this time, the method presented in [59] is slightly better on average. We thus used the Microsoft Demosaicing Canon Dataset [38] and the Microsoft Demosaicing Panasonic Dataset [38] to further compare our method with the one proposed in [59]. Table 5 presents the obtained average results on both datasets; the proposed algorithm always obtains the best results.
5.2 Noisy Images
In a second set of experiments, we considered noisy images corrupted by independent Gaussian noise, with zero mean and different standard deviation values \(\sigma \). This time, the best free parameters for stabilizer \(\varphi \) to be used in the second and third addend of the right hand of (4) are shown in Table 6 .
As done for the noiseless images, for each value of the noise variance, the suitable value \({\bar{p}}\) for stopping the algorithm has been determined according to the criterion established in (19). The following empirical law that relates \({\bar{p}}\) to \(\sigma \) has also been found:
In the noisy case, we compared our method with the algorithms in [29] and [34] by using the original MATLAB code provided by the authors and with the algorithms in [20, 76], and [55], by using the source codes available in the authors’ web pages. The CPSNR values computed for the case \(\sigma =16\) on the Kodak dataset are shown in Table 7.
Although the performance of our method is still satisfactory, this time, the method in [20] is slightly superior.
We then computed another quality index, sometimes used in the demosaicing problem, i.e., the SCIELAB metric. This metric indicates the percentage of color distortion between two images and accounts for the spatialcolor sensitivity of the human eye [73] [37]. Since it returns a pixelbypixel matrix of errors, we assumed the mean of the SCIELAB matrix coefficients as the representative error index for the entire image. The results obtained for the case \(\sigma =16\) on the Kodak dataset, along with the results of the most performing among the methods used for comparison, are shown in Table 8.
The results in this case are excellent. The situation is even better when the noisy mosaiced McMaster images are processed. The CPSNR results obtained for the same amount of noise (\(\sigma =16\)), along with the results of the most performing methods used for comparison on the Kodak dataset, are shown in Table 9.
It is apparent that this time, our method outperforms the other, with much higher values of CPSNR, which are only slightly lower than those we obtained in the noiseless case for the same dataset. This excellent performance can be ascribed once again to our very fine modeling of natural images in terms of local variations inside and between the color channels.
Furthermore, in Table 10, it is possible to observe the excellent behavior of the proposed method on the two Microsoft datasets, compared with the method in [59] and for different values of the standard deviation \(\sigma \) of the noise corrupting the data.
6 Conclusion
We approached the joint demosaicing and denoising of color images within a regularization framework, irrespective of the CFA employed to generate the data. A central feature of our method is adopting local image smoothness models that implicitly account for “soft” edges at low and highfrequency levels. Soft edges are essential for preventing the aliasing effect and preserving the fine details in the image, especially when noise is present. In our method, this is achieved by using stabilizers possessing two main features: on the one hand, we include derivatives up to the third degree; on the other hand, these derivatives are weighted through nonquadratic functions. A duality theorem ensures that the implicitly addressed edge fields are equivalent to geometrically consistent image discontinuities at each cycle of the iterative algorithm employed to find the regularized solution.
The experimentation over the Kodak 24image dataset, the McMaster (IMAX) 18image dataset, the Microsoft Demosaicing Canon 57image dataset, and the Microsoft Demosaicing Panasonic 500image dataset under the Bayer CFA, with a high level of noise, demonstrates the good performance of our method against some of the bestperforming demosaicing algorithms proposed so far.
Future developments involve experimenting with the algorithm on data from other CFAs and including the filter blurs that are known in this kind of application. For this latter issue, the extension of the algorithm is straightforward, since the data term is still convex, so the same approximations for the energy function can also be exploited.
References
Aelterman, J., Goossens, B., De Vylder, J., Pizurica, A., Philips, W.: Computationally Efficient Locally Adaptive Demosaicing of Color Filter Array Images Using the DualTree Complex Wavelet Packet Transform. PLoS ONE 8(5), 1–18 (2013)
Baek, M., Jeong, J.: Demosaicing algorithm using highorder interpolation with sobel operators. In: Proceedings of the World Congress on Engineering, WCE 2014, Lecture Notes in Engineering and Computer Science 1, pp. 521524 (2014)
Bai, C., Li, J., Lin, Z., Yu, J., Chen, Y.W.: Penrose demosaicking. IEEE Trans. Image Process. 24, 1672–1684 (2015)
Bayer, B. E.: Color imaging array. U.S. Patent. 3 971 065 (1976)
Beck, A.: Introduction to nonlinear optimization  Theory, Algorithms, and Applications with MATLAB. SIAM Mathematical Optimization, Philadelphia, PA, USA (2014)
Bedini, L., Gerace, I., Pepe, M., Salerno, E., Tonazzini, A.: Stochastic and deterministic algorithms for image reconstruction with implicitly referred discontinuities. Internal report n. r/2/85, pages 37, Istituto di Elaborazione della Informazione, C.N.R., Pisa (Italy), (1992)
Bedini, L., Gerace, I., Tonazzini, A.: A GNC algorithm for constrained image reconstruction with continuousvalued line processes. Pattern Recognit. Lett. 15(9), 907–918, (1994)
Blake, A.: Comparison of the efficiency of deterministic and stochastic algorithms for visual reconstruction. IEEE Trans. Pattern Anal. Mach. Intell. 11(1), 2–12 (1989)
Blake, A., Zissermann, A.: Visual Reconstruction. MIT Press, Cambridge, MA (1987)
Boccuto, A., Gerace, I.: Image reconstruction with a nonparallelism constraint. In: Proceedings of the International Workshop on Computational Intelligence for Multimedia Understanding, Reggio Calabria, Italy, 2728 October 2016, IEEE Conference Publications, pp. 1–5 (2016)
Boccuto, A., Gerace, I., Giorgetti, V.: A Blind Source Separation Technique for Document Restoration. SIAM J. Imaging Sci. 12(2), 1135–1162 (2019)
Boccuto, A., Gerace, I., Giorgetti, V.: A Graduated Non–Convexity Technique for Dealing Large Point Spread Funtion Appl. Sci. 2023, 13(10), pages 28 (2023)
Boccuto, A., Gerace, I., Giorgetti, V., Rinaldi, M.: A fast algorithm for the demosaicing problem concerning the bayer pattern. The Open Signal Proc. J. 6, 1–14 (2019)
Boccuto, A., Gerace, I., Martinelli, F.: HalfQuadratic Image Restoration with a NonParallelism Constraint. J. Math. Imaging Vis. 59(2), 270–295 (2017)
Boccuto, A., Gerace, I., Pucci, P.: Convex approximation technique for interacting line elements deblurring: a new approach. J. Math. Imaging Vis. 44(2), 168–184 (2012)
Brewster, M.E., Kannan, R.: Nonlinear successive overrelaxation. Numer. Math. 44(2), 3015–3019 (1984)
Buades, A., Coll, B., Morel, J.M., Sbert, C.: Selfsimilarity driven color demosaicking. IEEE Trans. Image Process. 18(6), 1192–1202 (2009)
Chung, K.H., Chan, Y.H.: Color demosaicing using variance of color differences. IEEE Trans. Image Process. 15(10), 2944–2955 (2006)
Chung, K.H., Chan, Y.H.: Lowcomplexity color demosaicing algorithm based on integrated gradients. J. Electron. Imaging 19(2), 1–15 (2010)
Condat, L.: A simple, fast and efficient approach to demoisaicking: Joint demosaicking and denoising. Proc. IEEE Int. Conf. Image Proc. 50, 905–908 (2010)
Condat, L., Mosaddegh, S.: Joint demosaicking and denoising by total variation minimization. In: Proceedings of the Image Processing (ICIP), 2012 19th IEEE International Conference on. IEEE; September 2012; Orlando, FL, USA, pp. 2781–2784
Cui, K., Jin, Z., Steinbach, E.: Color image demosaicking using a 3Stage convolutional neural network structure, Proc. 25th IEEE Int. Conf. Image Process. (ICIP), pp. 21772181, Oct. (2018)
Evangelopoulos, X., Brockmeier, A. J., Mu, T., Goulermas, J. Y.: A Graduated NonConvexity Relaxation for LargeScale Seriation. In: Proceedings of the 2017 SIAM International Conference on Data Mining, Houston, Texas (USA), pp. 462–470 (2017)
Ferradans, S., Bertalmío, M., Caselles, V.: Geometrybased demosaicking. IEEE Trans. Image Process. 18(3), 665–670 (2009)
Geman, S., Geman, D.: Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images. IEEE Trans. Pattern Anal. Machine Intell. 6, 721–740 (1984)
Geman, D., Reynolds, G.: Constrained restoration and the recovery of discontinuities. IEEE Trans. Pattern Anal. Mach. Intell. 14(3), 367–383 (1992)
Gharbi, M., Chaurasia, G., Paris, S., Durand, F.: Deep joint demosaicking and denoising. ACM Trans. Graph. 35(6), 1–12 (2016)
Giorgetti, V.: Illposed Problems in Computer Vision. Ph. D. Thesis, (2022)
Gunturk, B.K., Altunbasak, Y., Mersereau, R.: Colorplane interpolation using alternating projections. IEEE Trans. Image Process. 11, 997–1013 (2002)
Gunturk, B.K., Glotzbach, J., Altunbasak, Y., Schafer, R.W., Mersereau, R.M.: Demosaicking: Color Filter Array Interpolation. IEEE Signal Process. Magazine 22(1), 44–54 (2005)
Hamilton, J. F., Compton, J. T: Processing Color and Panchromatic Pixels. U.S. Patent 2007 0024879, Feb. (2007)
Hazan, E., Levy, K. Y., ShalevShwartz, S.: On Graduated Optimization for Stochastic NonConvex Problems. In: Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA, 2016. JMLR: W & CP 48, pp. 1–9 (2016)
Hirakawa, K., Parks, T.W.: Adaptive homogeneitydirected demosaicing algorithm. IEEE Trans. Image Process. 14(3), 360–369 (2005)
Hirakawa, K., Parks, T.W.: Joint Demosaicing and Denoising. IEEE Trans. Image Process. 15(8), 2146–2157 (2006)
Hirakawa, K., Wolfe, P.: Spatiospectral color filter array design for enhanced image fidelity. In: 2007 IEEE International Conference on Image Processing (ICIP), pp. 81–84 (2007)
Jin, Q., Guo, Y., Morel, J.M., Facciolo, G.: A Mathematical Analysis and Implementation of Residual Interpolation Demosaicking Algorithms. Image Processing On Line 11, 234–283 (2021). https://doi.org/10.5201/ipol.2021.358
Johnson, G.M., Fairchild, M.: A top down description of SCIELAB and CIEDE2000. Color. Res. Appl. 28, 425–435 (2003)
Khashabi, D., Nowozin, S., Jancsary, J., Fitzgibbon, A.W.: Joint demosaicing and denoising via learned nonparametric random fields. IEEE Trans. Image Process. 23(12), 4968–4981 (2014)
Kiku, D., Monno, Y., Tanaka, M., Okutomi, M.: Residual interpolation for color image demosaicking. In: Proceedings of the 2013 20th IEEE International Conference on Image Processing (ICIP), Melbourne, VIC, Australia, pp. 2304?2308 (2013)
Kiku, D., Monno, Y., Tanaka, M., Okutomi, M.: MinimizedLaplacian residual interpolation for color image demosaicking. In: Proceedings of the SPIE, San Francisco, CA, USA, p. 90230L (2014)
Kiku, D., Monno, Y., Tanaka, M., Okutomi, M.: Beyond color difference: Residual interpolation for color image demosaicking. IEEE Trans. Image Process. 25, 1288–1300 (2016)
Klatzer, T., Hammernik, K., Knobelreiter, P., Pock, T.: Learning joint demosaicing and denoising based on sequential energy minimization. In: Proceedings of the Computational Photography (ICCP), 2016 IEEE International Conference on. IEEE; May; Evanston, IL, USA. pp. 1–11 (2016)
Kodak Eastman Company, “PhotoCD PCD0992”, (http://r0k.us/graphics/kodak/)
Kokkinos, F., Lefkimmiatis, S.: Iterative joint image demosaicking and denoising using a residual denoising network. IEEE Trans. Image Process. 28(8), 4177–4188 (2019)
Li, X.: Demosaicing by successive approximation. IEEE Trans. Image Process. 14(3), 370–379 (2005)
Li, X., Gunturk, B. K., Zhang, L.: Image demosaicing: A systematic survey. In: Proc. SPIEIS &T Electronic Imaging, Visual Communications and Image Processing, Jan, vol. 6822, no. 1, pp. 1–15 (2008)
Liu, L., Jia, X., Liu, J., Tian, Q.: Joint demosaicing and denoising with self guidance. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; August; Seattle, WA, USA. pp. 22402249 (2020)
Liu, Z.Y., Qiao, H.: GNCCPGraduated NonConvexity and Concavity Procedure. IEEE Trans. Pattern Anal. Mach. Intell. 36(6), 1258–1267 (2014)
Liu, Z.Y., Qiao, H., Su, J.–H: MAP Inference with MRF by Graduated NonConvexity and Concavity Procedure. In: Loo, C. K.; Yap, K. S.; Wong, K. W.; Teoh, A.; Huang, K. (Eds): Neural Information Processing. ICONIP 2014. Lecture Notes in Computer Science, 8835, 404–412 (2014)
Mairal, J., Elad, M., Sapiro, G.: Sparse Representation for Color Image Restoration. IEEE Trans. Image Process. 17(1), 53–69 (2008)
Martinec, E. J.: https://github.com/ethiccinema/mlrawviewer
Menon, D., Andriani, S., Calvagno, G.: Demosaicing with directional filtering and a posteriori decision. IEEE Trans. Image Process. 16(1), 132–141 (2007)
Menon, D., Calvagno, G.: Demosaicing based on wavelet analysis of the luminance component. Proc. IEEE Int. Conf. Image Processing 2, 181–184 (2007)
Menon, D., Calvagno, G.: Regularization Approaches to Demosaicking. IEEE Trans. Image Process. 18, 2209–2220 (2009)
Menon, D., Calvagno, G.: Joint demosaicking and denoising with spacevarying filters. In: Proceedings of the 2009 16th IEEE International Conference on Image Processing (ICIP), pp. 477–480 (2009)
Menon, D., Calvagno, G.: Color image demosaicking  An overview. Signal Processing: Image Communication 26, 518–533 (2011)
Mobahi, H., Fisher, J.W.: A Theoretical Analysis of Optimization by Gaussian Continuation. In: Wong, W.K., Lowd, D. (eds.) TwentyNinth Conference on Artificial Intelligence of the Association for the Advancement of Artificial Intelligence (AAAI), Proceedings, pp. 1205–1211. Austin, Texas, USA, January 25–30(2015) (2015)
Moghadam, A., Aghagolzadeh, M., Kumar, M., Radha, R.: Compressive framework for demosaicing of natural images. IEEE Trans. Image Process. 22, 2356–2371 (2013)
Monno, Y., Kiku, D., Tanaka, M., Okutomi, M.: Adaptive residual interpolation for color and multispectral image demosaicking. Sensors 17(12), 2787 (2017)
Nikolova, M.: Markovian reconstruction using a GNC approach. IEEE Trans. Image Process. 8(9), 1204–1220 (1999)
Nikolova, M., Ng, M.K., Tam, C.P.: On \(\ell _1\) Data Fitting and Concave Regularization for Image Recovery. SIAM J. Sci. Comput. 35(1), A397–A430 (2013)
Nikolova, M., Ng, M.K., Zhang, S., Ching, W.K.: Efficient Reconstruction of Piecewise Constant Images Using Nonsmooth Nonconvex Minimization. SIAM J. Imaging Sci. 1(1), 2–25 (2008)
Park, Y., Lee, S., Jeong, B., Yoon, J.: Joint demosaicing and denoising based on a variational deep image prior neural network. Sensors 20(10), 2970 (2020)
RawTherapee Program, https://rawpedia.rawtherapee.com
Robini, M.C., Magnin, I.E.: Optimization by Stochastic Continuation. SIAM J. Imaging Sci. 3(4), 1096–1121 (2010)
Saito, T., Komatsu, T.: Demosaicing approach based on extended color totalvariation regularization. In: Proc. IEEE Int. Conf. Image Processing, Sep, vol. 1, pp. 885–888 (2008)
Smith, T., Egeland, O.: Dynamical Pose Estimation with Graduated NonConvexity for Outlier Robustness. MICModel. Identif. Control 43(2), 79–89 (2022)
Tan, D.S., Chen, W.Y., Hua, K.L.: DeepDemosaicking: Adaptive image demosaicking via multiple deep fully convolutional networks. IEEE Trans. Image Process. 27(5), 2408–2419 (2018)
Tan, R., Zhang, K., Zuo, W., Zhang, L.: Color image demosaicking via deep residual learning. In: Proceedings of the IEEE Int. Conf. Multimedia Expo (ICME), pp. 793–798 (2017)
Xing, W., Egiazarian, K.: Endtoend learning for joint image demosaicing, denoising and superresolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; June; Nashville, TN, USA. pp. 3507–3516 (2021)
Yang, H., Antonante, P., Tzoumas, V., Carlone, L.: Graduated nonconvexity for robust spatial perception: from nonminimal solvers to global outlier rejection. IEEE Robot. Auto. Lett. 5(2), 1127–1134 (2020)
Ye, W., Ma, K.K.: Color image demosaicing using iterative residual interpolation. IEEE Trans. Image Process. 24(12), 5879–5891 (2015)
Zhang, X., Wandell, B.: A spatial extension of CIELAB for digital colorimage reproduction. J. Soc. Inf. Disp. 5(1), 61–63 (1997)
Zhang, L., Wu, X.: Color demosaicking via directional linear minimum mean squareerror estimation. IEEE Trans. Image Process. 14(12), 2167–2177 (2005)
Zhang, L., Wu, X., Buades, A., Li, X.: Color Demosaicking by Local Directional Interpolation and Nonlocal Adaptive Thresholding. J. of Electronic Imaging 20 (2), 023016, pp. 1–17 (2011)
Zhang, L., Wu, X., Zhang, D.: Color reproduction from noisy CFA data of single sensor digital cameras. IEEE Trans. Image Process. 16(9), 2184–2197 (2007)
Funding
Open access funding provided by Universitá degli Studi di Perugia within the CRUICARE Agreement.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix A: Geometry of the Cliques and Expression of the Associated Finite Differences
The stabilizers used in this paper are functions of the finite differences \(D_c^k\) of order k applied to sets c consisting of adjacent pixels. We call clique of order k the set of pixels on which the finite difference of order k is well defined. We take \(k=1,2,3\) in order to reconstruct the finest details in images. Figures 4, 5 and 6 show the geometry of the sets c for the three orders of finite differences, respectively. As we can see, the cliques can be classified as vertical (Figs. 4 (a), 5 (a), 6 (a) ), horizontal (Figs. 4 (b), 5 (c), 6 (d) ), and mixed (Figs. 5 (b), 6 (b) and (c) ).
The vertical cliques consist of the following pixels:
while the horizontal cliques have the form
Let us now describe how finite differences are computed at a generic clique c for a generic color channel \(\textbf{x}^{(e)}\), \(e \in \{r,g,b\}\). When \(k=1\), as it is seen in Fig. 4, we have two different kinds of finite difference operators, associated with a horizontal and a vertical finite difference, given by
respectively. When \(k=2\), we have three different kinds of finite difference operators, expressed by
When \(k=3\), we get four different kinds of finite difference operators, given by
Let us introduce the concept of adjacent clique of order k, which defines the nonparallelism constraint. Given a vertical clique
\(c=\{(i,j), (i+1,j), \ldots , (i+k,j) \}\), \(i=k+1, \ldots , nk\), \(j=1,\ldots , m\), \(k=1,2,3\),
we define its preceding clique \(p_k(c)\) as follows:
When \(i=k+2, \ldots , nk\), \(j=1,\ldots , m\), \(k=1,2,3\), a good approximation of \(p_k(c)\) used to construct our approximating functions is given by
We define \(\pi _k(c)\) in such a way that \(c \cap \pi _k(c)= \emptyset \). This will be useful to find the family of approximations of the energy function in Subsection 3.2.
If c is a horizontal clique, \(c=\{(i,j), (i,j+1), \ldots , (i,j+k) \}\), \(i=1, \ldots , n\), \(j=k+1,\ldots , mk\), \(k=1,2,3\), then its preceding clique \(p_k(c)\) is defined by
When \(i=1, \ldots , n\), \(j=k+2,\ldots , mk\), \(k=1,2,3\), a good approximation of \(p_k(c)\) is
For mixed cliques and cliques on the board of the image, \(p_k(c)\) and \(\pi _k(c)\) are considered not to be defined.
Appendix B: Duality Conditions on the Stabilizer
In order for a stabilizer \(\varphi \) to be edgepreserving and that the nonparallelism constraint on the implicit line process is satisfied, we require that the hypotheses of the following theorem are satisfied (see [14]):
Theorem 1
For every \(p \in [1,2]\), let
where \({\overline{t}}= 2^k \cdot \sqrt{2} \cdot 255\), \(k=1,2,3\), for light intensity of the images in the range [0, 255], is the maximum value which the variable \(t_2\) can assume, \(\overline{\varphi }\) and \(\varphi \) are as in (14) and (7), respectively. Then, \(\varphi ^{(p)}\) satisfies the following conditions:
 H1):

for every \(t_2 \in [{\overline{t}}, {\overline{t}}]\), the function \(\varphi _{t_2}: {\mathbb {R}} \rightarrow {\mathbb {R}} \cup \{  \infty \}\) defined by \(\varphi _{t_2}(t_1)=\varphi (t_1, t_2)\) is upper semicontinuous and even on \({\mathbb {R}}\), and \(\varphi _{t_2}(0)\in {\mathbb {R}}\);
 H2):

for each \(t_2 \in [{\overline{t}}, {\overline{t}}]\), the function \(\psi _{t_2}: {\mathbb {R}} \rightarrow {\mathbb {R}} \cup \{  \infty \}\) defined by
$$\begin{aligned} \psi _{t_2}(t_1)= \left\{ \begin{array}{ll}\varphi (\sqrt{t_1}, t_2), &{} \text { if } t_1 \ge 0,\\ \infty , &{} \text { if } t_1 < 0, \end{array}\right. \end{aligned}$$is concave on \({\mathbb {R}}^+_0\);
 H3):

\(\varphi _{t_2}\) is nondecreasing on \({\mathbb {R}}^+_0\) for every \(t_2 \in [{\overline{t}}, {\overline{t}}]\);
 H4):

\(\displaystyle {\lim _{t_1\rightarrow + \infty } \dfrac{\psi _{t_2}(t_1)}{t_1} =0}\) for each \(t_2 \in [{\overline{t}}, {\overline{t}}]\),
 H5):

there exists at least a real number \(t_1\) such that the function \(\varphi _{t_1}(t_2)=\varphi (t_1, t_2)\) is not constant on \([{\overline{t}}, {\overline{t}}]\), and \(\varphi _{t_1}\) is even on \([{\overline{t}}, {\overline{t}}]\) and nondecreasing on \([0, {\overline{t}}]\) for every \(t_1 \in {\mathbb {R}}^+_0\).
Proof
We begin by proving that the function \(\overline{\varphi }\) defined in (14) satisfies Hj), \(j=1, \ldots , 4\).
It is readily seen that \(\overline{\varphi }\) fulfills H1).
Now we prove H2).
For \(i=1,2\) and \(t_1 \in {\mathbb {R}}^+_0\), set
We have
and hence
Let \(\displaystyle {\gamma _{t_2}= \dfrac{{\overline{t}}^2t_2^2}{{\overline{t}}^2}}\). Then, \(1\gamma _{t_2}=\dfrac{t_2^2}{{\overline{t}}^2}\).
It is not difficult to check that \(0 \le \gamma _{t_2} \le 1\), since \(t_2 \le {\overline{t}}\). Thus, for every \(t_1 \in {\mathbb {R}}^+_0\) and \(t_1 \ne q_1^2\), \(t_1 \ne q_2^2\), \(t_2 \in [{\overline{t}}, {\overline{t}}]\), we have
Observe that the inequality in (B8) will be useful to prove H3). Since \({\overline{\psi }}_{t_2}\) is of class \(C^1\) on its domain (indeed, it is a composition of \(C^1\) functions), then it is concave on \({\mathbb {R}}_0^+\) for all \(t_2 \in [{\overline{t}}, {\overline{t}}]\). So, \(\overline{\varphi }\) satisfies condition H2).
Now we prove H3). From (B8), it follows that \({\overline{\psi }}_{t_2}\) is nondecreasing on \({\mathbb {R}}^+_0\), and hence so is \({\overline{\varphi }}_{t_2}\).
Thus we get
for each \(t_1 \in {\mathbb {R}}^+_0\). Thus, H3) holds.
Now we show that \(\overline{\varphi }\) fulfils H4). Indeed, we have
and hence
Finally, we prove that \(\overline{\varphi }\) satisfies H5).
Take \(t_1=q_2\). For each \(t_2 \in [{\overline{t}}, {\overline{t}}]\), it is
and hence \(\overline{\varphi }(q_2,0) =\lambda ^2 \, ( 2 \, q_1 \, q_2  q_1^2)\), \(\overline{\varphi }(q_2,{\overline{t}})=\lambda ^2 \, q_2^2\). We claim that \(\overline{\varphi }(q_2,0) \ne \overline{\varphi }(q_2,{\overline{t}})\). If not, then we would have \(2 \, q_1 \, q_2  q_1^2 = q_2^2\), and hence \(0=q_2^2  2 \, q_1 \, q_2 + q_1^2 = (q_1  q_2)^2\), that is \(q_1=q_2\), which is absurd since we know that \(0< q_1 < q_2\). Therefore, the function \(t_2 \mapsto \overline{\varphi }(q_2, t_2)\) is not constant, and hence, the first property of H5) is satisfied.
Moreover, it is easy to see that \({\overline{\varphi }}_{t_1}\) is even on \({\mathbb {R}}\) for each \(t_1 \in {\mathbb {R}}\).
From (13), it is not difficult to deduce that \(\overline{g_2}(t_1)  \overline{g_1} (t_1) \ge 0\) for all \(t_1 \ge 0\).
We get
for any \(t_1 \in {\mathbb {R}}^+_0\). Hence, the function \(\overline{\varphi }_{t_1}\) is nondecreasing on \({\mathbb {R}}^+_0\) for every \(t_1 \in {\mathbb {R}}^+_0\). Thus, H5) is proved.
Now we prove that for \(i=1,2\), the function \(\varphi \) defined in (7) satisfies conditions Hj), \(j=1, \ldots , 4\).
It is easy to see that, by construction, H1) holds.
We now prove H2). We begin with the case when \(t_2 \le s\) or \(t_2 \ge \zeta \) in (7). For \(i=1,2\), set
We have
We claim that \(f_i\) is nondecreasing and concave on \({\mathbb {R}}^+_0\). We get
Note that \(f_i\) is \(C^1\), since it is a composition of functions of class \(C^1\). Moreover, we have
From this, we deduce that \(\varphi \) fulfils H2), at least when \(t_2 \le s\) or \(t_2 \ge \zeta \).
Now, we examine the case
We have
Observe that \(\dfrac{2(t_2 s)^2}{(\zeta s)^2}\ge 0\). Now, we claim that \(\dfrac{2(t_2 s)^2}{(\zeta s)^2}\le 1\). Indeed, since \(s< t_2 \le \dfrac{\zeta +s}{2}\), then \(0<t_2 s \le \dfrac{\zeta s}{2} < \dfrac{\zeta s}{\sqrt{2}}\), and hence \((t_2 s)^2 \le \dfrac{(\zeta s)^2}{2}\), getting the claim. Therefore, \(1 \dfrac{2(t_2 s)^2}{(\zeta s)^2} \ge 0\). From this, since \(f_i^{\prime }(t_1) \ge 0\) for every \(t_1 \ge 0\) and \(f_i^{\prime \prime }(t_1) \le 0\) for each \(t_1 {\mathbb {R}}^+_0\), \(t_1 \ne q_1\), \(t_1 \ne q_2\), in the case (B14) we obtain
Note that the inequality in (B17) will be useful to prove H3).
From (B18), taking into account the continuity of \(\psi _{t_2}\), we deduce that \(\varphi \) satisfies H2) also in the case (B14).
Now, we consider the case
We get
Note that \(\dfrac{2(t_2 \zeta )^2}{(\zeta s)^2}\ge 0\). Now we claim that \(\dfrac{2(t_2 \zeta )^2}{(\zeta s)^2}\le 1\). Indeed, as \(\dfrac{\zeta +s}{2}< t_2 < \zeta \), then \(0<\zeta  t_2 \le \dfrac{\zeta s}{2} < \dfrac{\zeta s}{\sqrt{2}}\), and so \((t_2 \zeta )^2 \le \dfrac{(\zeta s)^2}{2}\), getting the claim. Thus, \(1 \dfrac{2(t_2 \zeta )^2}{(\zeta s)^2} \ge 0\). From this, in the case (B19), analogously as in the case (B14), we obtain
From (B23) and thanks to the continuity of \(\psi _{t_2}\), it follows that \(\varphi \) satisfies H2) also in the case (B19).
Now we prove that \(\varphi \) satisfies H3).
First, when \(t_2 \le s\) or \(t_2 \ge \zeta \), observe that it is readily seen that \(g_i\) is nondecreasing on \({\mathbb {R}}^+_0\) for \(i=1,2\). Hence, \(\psi _{t_2}\) is nondecreasing on \({\mathbb {R}}^+_0\), and thus H3) holds.
Moreover, when \(t_2\) satisfies the case (B14) or the case (B19), from (B17) and (B22) it follows that \(\psi _{t_2}\) is nondecreasing on \({\mathbb {R}}^+_0\), and hence \(\varphi _{t_2}\) is too. Thus, \(\varphi \) fulfils H3).
Now we prove H4). That \(\varphi \) satisfies H4) is a consequence of the fact that \(\displaystyle {\lim _{t_1\rightarrow + \infty } \dfrac{f_i(t_1)}{t_1} =0}\).
Now we prove H5). Let
First of all, observe that \(\varphi (r_2,s)=g_1(r_2)=\alpha \), \(\varphi (r_2,\zeta )=g_2(r_2)=\alpha +\varepsilon \ne \varphi (r_2,s)\). Thus, the function \(t_2\mapsto \varphi (r_2, t_2)\) is not constant. Moreover, it is easy to see that the function \(\varphi _{t_1}(t_2)\) is even on \([{\overline{t}}, {\overline{t}}]\) for every \(t_1 \in {\mathbb {R}}^+_0\), since it depends on \(t_2 \). Furthermore, it is not difficult to check that
Let \(t_2 \in [0, {\overline{t}}]\). We get
in the case (B14), and
in the case (B19). From (B24), (B25) and (B26) it follows that \(\varphi _{t_1}^{\prime }(t_2)\ge 0\) for each \(t_2 \in [0, {\overline{t}}]\). By arbitrariness of \(t_1 \in {\mathbb {R}}^+_0\), we deduce that \(\varphi \) satisfies H5).
Now, we observe that the functions \(\varphi ^{(p)}\), \(p \in [0,2]\), satisfy conditions Hj), \(j=1, \ldots , 4\), since they are nonnegative linear combinations of functions satisfying Hj), \(j=1, \ldots , 4\). Since \(\overline{\varphi }_{t_1}\) and \(\varphi _{t_1}\) are nondecreasing for each \(t_1 \in {\mathbb {R}}^+_0\), \(\overline{\varphi }_{t_2}\) and \(\varphi _{t_2}\) are nondecreasing for every \(t_2 \in [{\overline{t}}, {\overline{t}}]\), and the functions \(t_2 \mapsto \overline{\varphi }(q_2,t_2)\), \(t_2 \mapsto \varphi (r_2,t_2)\) are not constant, it follows that for every \(p \in [0,2]\) there exists at least a \(t_1 \in {\mathbb {R}}^+_0\) such that the function \(t_2 \mapsto \varphi ^{(p)}(t_1, t_2)\) is not constant. The other properties of H5) hold because the \(\varphi ^{(p)}\)’s are nonnegative linear combinations of functions satisfying H5). \(\square \)
Appendix C: Convergence of the NLSOR Algorithm
To minimize each approximation \(E^{(p)}\), \(p \in [0,2]\), we use the NLSOR algorithm, described in Subsection 4.1.
We will prove the existence of suitable limit points, which are stationary points of \(E^{(0)}\) (in general, they are not minimum points of \(E^{(0)})\). In [16, Theorem 2] the convergence of the algorithm is proved when \(E^{(0)}\) is strictly convex and of class \(C^2\). Such assumptions are too strong for the componentwise convex approximation of the regularization term in our setting because we deal with functions of class \(C^1\). So, we give an extension of the theorem under these weaker hypotheses.
First, we state the following technical lemma, whose proof is given in [28, Lemma 2.7.1].
Lemma 2
Let \(\phi :[x_0,{\overline{x}}] \rightarrow {\mathbb {R}}\) be convex, of class \(C^1\), having both left and right second derivative on \([x_0,{\overline{x}}]\). Suppose that \(\phi \) is second differentiable on \([x_0, {\overline{x}}] \setminus P\), where \(P=\{ x_j: j=1, \ldots , N \}\), with \(x_0< x_1< \ldots< x_N < {\overline{x}}\).
Then, for every \(x \in [x_0,{\overline{x}}]\) there exist \(\xi \in ]x_0,x[\) and \(\mu \ge 0\), such that
and
Observe that when \(p=0\), the NLSOR can be formulated as in Algorithm 3. At the iterate \(l \in {\mathbb {N}}\), fixed \(i \in {1,2, \ldots , 3\,m}\) and \(e \in \{r,g,b\}\), the vector \(\textbf{x}^{(l, i, e)}\) is defined by
where
Algorithm 3 allows to denote the image vectors actually defined at each iterate l, at every pixel i and at each color e. We observe that the algorithm here proposed is a particular case of that given in Subsection 4.1, and has been suitably modified in order to give a rigorous definition of the image vector \({\textbf{x}}\) at every iterate l, at every pixel i and at each color e.
Moreover, fixed the step (l, i, e), let \({\textbf{x}^{(l, i, e)} {\setminus } (x^{(l,i,e)})}_i^{(e)} \in {\mathbb {R}}^{3nm1}\) be the vector whose elements are those of \({\textbf{x}}^{(l, i, e)}\) except \({(x^{(l,i,e))})}_i^{(e)}\). The value of this pixel \({(x^{(l,i,e))})}_i^{(e)}\) is an unknown variable, which we call z. For each fixed value of \({\textbf{x}}^{(l, i, e)} {\setminus } {(x^{(l,i,e))})}_i^{(e)}\), let us define the following energy function \({\overline{E}}^{(l,i,e)}:{\mathbb {R}} \rightarrow {\mathbb {R}}\) by
Theorem 3
Let \(E^{(0)}:{\mathbb {R}}^{3nm} \rightarrow {\mathbb {R}}\) be a function of class \(C^1\) and coercive, that is
fix \({{\textbf{x}}}^{(0, nm,b)} \in {\mathbb {R}}^{3nm}\), and let \(\{ {{\textbf{x}}}^{(l,i,e)}\}\), \(l \in {\mathbb {N}}, i=1,2,\ldots , 3\,m, e \in \{r,g,b\}\), be the sequence defined iteratively in (C29), where \(0< \omega < 2\) and
Let \({\overline{E}}^{(l,i,e)}:{\mathbb {R}} \rightarrow {\mathbb {R}}\) be the function defined in (C31). Assume that \({\overline{E}}^{(l,i,e)}\) is convex, admits both left and right derivative on \({\mathbb {R}}\) and is not secondly differentiable (at most) at a finite number of points.
Then, \(\displaystyle { \lim _{(l,i,e)} \nabla E^{(0)} (x^{(l,i,e)})=0}\).
Proof
We begin by proving that, during the updating of \({(x^{(l,i,e))})}_i^{(e)}\), the function \(E^{(0)}\) is nonincreasing.
If
then, since in (C29) it is \(\displaystyle {{(x^{\text {prec}(l,i,e))})}_i^{(e)}= \dfrac{\displaystyle {\partial E^{(0)}} \bigl (x^{\text {prec}(l, i, e)} \bigr )}{\partial x_i^{(e)}} }\), we get
and hence \(\textbf{x}^{(l,i,e)}=\textbf{x}^{\text {prec}(l,i,e)}\), Moreover, the value of the energy function does not change.
Now, we treat the case when
Note that, by (C32), we get
that is the function \({\overline{E}}^{(l,i,e)}\) is coercive on \({\mathbb {R}}\). Since \({\overline{E}}^{(l,i,e)}\) is also continuous, then, by [5, Theorem 2.32], \({\overline{E}}^{(l,i,e)}\) assumes the minimum value, say \({(t_*)}^{(l,i,e)}\).
We get that, for any \(t > {(t_*)}^{(l,i,e)}\), the level set \(L_t=\{z \in {\mathbb {R}}: {\overline{E}}^{(l,i,e)}(z) =t \}\) has exactly two points, and \({\overline{E}}^{(l,i,e)}(z)<t\) whenever z is in the interior of the interval whose endpoints are the elements of \(L_t\).
Now we claim that, for every \(t > {(t_*)}^{(l,i,e)}\), the level set \(L_t=\{z \in {\mathbb {R}}: {\overline{E}}^{(l,i,e)}(z) = t^{(l,i,e)} \}\) has exactly two points.
Since \({\overline{E}}^{(l,i,e)}\) is convex and differentiable, we get that \({\overline{E}}^{(l,i,e)}(z)= {(t_*)}^{(l,i,e)}\) if and only if \({{\overline{E}}^{(l,i,e)}}^{\prime }(z)=0\). From the continuity of \({\overline{E}}^{(l,i,e)}\) and (C32) it follows that \({\overline{E}}^{(l,i,e)}\) assumes all values \(t \in [{(t_*)}^{(l,i,e)}, + \infty [\). Since \({{\overline{E}}^{(l,i,e)}}^{\prime }\) is nondecreasing, then \({{\overline{E}}^{(l,i,e)}}^{\prime }\) is positive (resp. negative), and hence \({\overline{E}}^{(l,i,e)}\) is strictly increasing (resp. decreasing) at all greater (resp. smaller) points than the minimum points of \({\overline{E}}^{(l,i,e)}\). Thus, \({\overline{E}}^{(l,i,e)}\) assumes each value \(t > {(t_*)}^{(l,i,e)}\) exactly two times, getting the claim.
Set \(t^{(l,i,e)}= {\overline{E}}^{(l,i,e)}({(x^{\text {prec}(l,i,e))})}_i^{(e)})\), and \(L_{t^{(l,i,e)}}= \{ {(x^{\text {prec}(l,i,e))})}_i^{(e)}, {\overline{z}}^{(l,i,e)}\}\), where \({\overline{E}}^{(l,i,e)}({\overline{z}}^{(l,i,e)})= t^{(l,i,e)}\).
Without loss of generality, let us consider the case \({{\overline{z}}}^{(l,i,e)}< {(x^{\text {prec}(l,i,e))})}_i^{(e)}\). Note that, in this case, \({{\overline{E}}^{(l,i,e)}}^{\prime } ({{\overline{z}}}^{(l,i,e)}) < 0\), while \({{\overline{E}}^{(l,i,e)}}^{\prime }({(x^{\text {prec}(l,i,e))})}_i^{(e)}) >0\).
By hypothesis, taking into account that \({\overline{E}}^{(l,i,e)}\) is of class \(C^1\), from Lemma 2 applied to the interval \([{{\overline{z}}}^{(l,i,e)}, {(x^{\text {prec}(l,i,e))})}_i^{(e)}]\) we find \(\xi \in ]{{\overline{z}}}^{(l,i,e)}, {(x^{\text {prec}(l,i,e))})}_i^{(e)}[\) and \(\mu \ge 0\), such that
and
Since \({\overline{E}}^{(l,i,e)}({(x^{\text {prec}(l,i,e))})}_i^{(e)})={\overline{E}}^{(l,i,e)} ({{\overline{z}}}^{(l,i,e)})\), then we have
Note that from (C38) and (C33) we get
Now we claim that \(\mu >0\). Indeed, if \(\mu =0\), then from (C39) we get
and hence \({{\overline{z}}}^{(l,i,e)} = {(x^{\text {prec}(l,i,e))})}_i^{(e)}\), because \({{\overline{E}}^{(l,i,e)}}^{\prime } ({(x^{\text {prec}(l,i,e))})}_i^{(e)})>0\). This is absurd because \({{\overline{z}}}^{(l,i,e)} < {(x^{\text {prec}(l,i,e))}}_i^{(e)}\). Therefore, we get the claim. From (C39), we obtain
We recall that, by (C29), it is
Since \(0< \omega < 2\), from (C40), (C41) and (C42) we have
From (C43) it follows that
Analogously, it is possible to prove that
Thus, in the case (C36), we get
\({\overline{E}}^{(l,i,e)}({(x^{(l,i,e))})}_i^{(e)}) < {\overline{E}}^{(l,i,e)}({(x^{\text {prec}(l,i,e))})}_i^{(e)})\). Therefore, in both cases (C34) and (C36), the sequence \(\{{\overline{E}}^{(l,i,e)}({(x^{(l,i,e))})}_i^{(e)})\}\), \(l \in {\mathbb {N}}\), \(i=1,2, \ldots , nm\), \(e \in \{r,g,b\}\), is nonincreasing. Since \(E^{(0)}\) is bounded from below, then the sequence \(\{{\overline{E}}^{(l,i,e)}({(x^{(l,i,e))})}_i^{(e)})\}\), \(l \in {\mathbb {N}}\), \(i=1,2, \ldots , nm\), \(e \in \{r,g,b\}\), is nonincreasing, and hence it is convergent.
Now, we claim that the sequence \(\{{(x^{(l,i,e))})}_i^{(e)}{(x^{\text {prec}(l,i,e))})}_i^{(e)} \}\), \(l \in {\mathbb {N}}\), \(i=1,2, \ldots , nm\), \(e \in \{r,g,b\}\), converges to 0.
Fix \(l \in {\mathbb {N}}\), \(i=1,2, \ldots , nm\), \(e \in \{r,g,b\}\). If \({{\overline{E}}^{(l,i,e)}}^{\prime } ({(x^{\text {prec}(l,i,e))})}_i^{(e)})= 0\), then, as seen in (C35), we get \({(x^{(l,i,e))})}_i^{(e)}={(x^{\text {prec}(l,i,e))})}_i^{(e)}\).
Now, we consider the case when \({{\overline{E}}^{(l,i,e)}}^{\prime } ({(x^{\text {prec}(l,i,e))})}_i^{(e)}) \ne 0\). By an argument similar to that used in the proof of [16, Theorem 2], from Lemma 2 applied to the interval whose endpoints are \({(x^{(l,i,e))})}_i^{(e)}\) and \({(x^{\text {prec}(l,i,e))})}_i^{(e)}\), we find a nonnegative real number \(\mu \) with
From (C42) we get
From (C46) and (C47) we obtain
As \(0< \omega < 2\) and \(0 \le \mu < T\), we get
From (C48) and (C49) we obtain
Note that (C50) holds also when \({{\overline{E}}^{(l,i,e)}}^{\prime } ({(x^{\text {prec}(l,i,e))})}_i^{(e)}) = 0\). Thus, in both cases (C34) and (C36), from (C50) and the convergence of the sequence \(\{{\overline{E}}^{(l,i,e)}({(x^{(l,i,e))})}_i^{(e)})\}\), \(l \in {\mathbb {N}}\), \(i=1,2, \ldots , nm\), \(e \in \{r,g,b\}\), it follows that the sequence \(\{ ( {\overline{E}}^{(l,i,e)}({(x^{\text {prec}(l,i,e))})}_i^{(e)}) {\overline{E}}^{(l,i,e)}({(x^{(l,i,e))})}_i^{(e)}) \,) \}\), \(l \in {\mathbb {N}}\), \(i=1,2, \ldots , nm\), \(e \in \{r,g,b\}\), converges to 0. From this, it follows that
getting the claim.
From (C29) we deduce
By arbitrariness of \( i \in \{ 1,2, \ldots , nm\}\) and \(e \in \{ r, g, b \}\), from (C51) and (C52) we deduce that \(\displaystyle {\lim _{(l,i,e)} \nabla E^{(0)} (x^{(l,i,e)})}\displaystyle {=0}\), that is the assertion. \(\square \)
Appendix D: Componentwise Convexity of the First Approximation
Now, we prove that the first approximation is componentwise convex.
Theorem 4
When \(p=0\), the function \(E^{(p)}\) in (15) is componentwise convex.
Proof
We recall that \(\overline{\varphi }\) is componentwise convex on \({\mathbb {R}}^2\) with respect to \(t_1\) and \(t_2\). Fix \(k \in \{1,2,3\}\) and \(c \in C_k\), and choose \(\Xi ^k_c \in \{N^k_c,V^k_c\}\).
Now we claim that the function \(\textbf{x} \mapsto \overline{\varphi }(\Xi _c^k \textbf{x}, \Xi _{\pi _k(c)} \textbf{x})\) is componentwise convex concerning the components of \(\textbf{x}\in {\mathbb {R}}^{3nm}\). Indeed, fix \(x_{i,j}^{(e)}\) with \(i \in \{1,2,\ldots , n\}\), \(j \in \{1,2,\ldots , m\}\) and \(e \in \{r,g,b \}\).
Now we prove the convexity of \(\overline{\varphi }\) with respect to the variable \(x_{i,j}^ {(e)}\), in the following three cases:
 I):

\((i,j ) \not \in c \cup \pi _k(c)\);
 II):

\((i,j ) \in c \);
 III):

\((i,j ) \in \pi _k(c)\).
We observe that it is impossible that \((i,j) \in c \cap \pi _k(c)\), thanks to our definition of \(\pi _k(c)\).
Fix arbitrarily \(u,w \in {\mathbb {R}}\) and \(t \in [0,1]\).
In case I), note that \(\Xi _c^k \textbf{x}\) and \(\Xi ^k_{\pi _k(c)}\textbf{x}\) are independent of the value of the variable \(x_{i,j}^{(e)}\). So, we get the function \(\overline{\varphi }\) evaluated in \(\Xi _c^k\), where all pixels are fixed except \(x_{i,j}^{(e)}\).
Fixed an image \(\textbf{x}\), let \(\textbf{x} {\setminus } x_{i,j}^{(e)} \in {\mathbb {R}}^{3nm1}\) be the vector whose elements are those of \({\textbf{x}}\) with the exception of \(x_{i,j}^{(e)}\). Observe that the value of this pixel \(x_{i,j}^{(e)}\) is an unknown variable. We have:
since
for each \( a, b \in {\mathbb {R}}\).
Now we deal with the case II).
It is not difficult to see that since the finite difference operators \(D_c^k\) are linear and the norm \(\Vert \cdot \Vert _2\) is a convex function, the operators \(\Xi _c^k\) and \(\Xi _{\pi _k(c)}\) are convex on their domain. We get
The first inequality in (D55) holds, since \(\Xi _c^k\) is (globally) convex. Note that \(\overline{\varphi }\) is increasing in the first component.
Furthermore, the third inequality in (D55) follows from (D54), since the function \((t_1,t_2) \mapsto \overline{\varphi } (t_1,t_2)\) is componentwise convex with respect to the variables \(t_1\) and \(t_2\), and since \((i,j) \not \in \pi _k(c)\).
Case III) is analogous to the case II). Thus, the assertion follows.
\(\square \)
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Boccuto, A., Gerace, I., Giorgetti, V. et al. An EdgePreserving Regularization Model for the Demosaicing of Noisy Color Images. J Math Imaging Vis 66, 904–925 (2024). https://doi.org/10.1007/s1085102401204y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s1085102401204y