Keywords

1 Introduction

In a standard problem of gray scale image denosing problem, the noisy image \(u_0\) corrupted by additive white Gaussian noise is modeled as

$$\begin{aligned} u_{0}(x,y)=u(x,y)+\sigma (x,y), \end{aligned}$$
(1)

where u is the unknown noisy free image and \(\sigma \) is assumed as known noise level: \(\int _{\varOmega }(u-u_{0})^{2}dxdy = \sigma ^{2}\). The goal of image restoration is to remove noise while preserving the important structure features from the observed noisy image \(u_0\) [1]. An usual regularization approach to remove noise by minimizing the following functional:

$$\begin{aligned} E(u,\lambda )= E{(u)} + \frac{\lambda }{2} \int _{\varOmega }(u-u_{0})^{2}dxdy, \end{aligned}$$
(2)

where E(u) is the regularization term to measure the variation of the noise intensity and \(\lambda \ge 0\) is the Lagrange multiplier. The first regularization term on the right-hand side of Eq. (2) is to measure the oscillations using weighted Laplace operator. The second fitting term is to measure the identification between u and \(u_{0}\). In seminar total variational (TV) method [2], the regularization functional is defined as

$$\begin{aligned} E_{\text {TV}}(u)=\int _{\varOmega }|\nabla u|dxdy, \end{aligned}$$
(3)

which produces a piecewise constant image while removing noise. However, TV suffers from staircase effect in smoothing transition region [3]. A more smoothing image is also expected in varying image processing fields, inluding computer photography [4], medical image processing [5], image registration [6], Retinex problem [7]. Some operations have been used to construct high order models, such as Laplace operation based YK model [3] and LLT model [5], the Frobenius norm of the Hessian based affine TV model [8], curvature based elastic model [9] mean curvature based model [10] and Gaussian curvature based model [11]. A variable exponent high order variational model was proposed in [12], where the Gaussian convolution was used for detecting edges.

Low order model and high order operators are combined to construct new methods: one part to produce flat image and the other part to generate smoothing transition. In [15], Papafitsoros and Sch\(\ddot{\text {o}}\)nlieb considered a general additive high order functional and proved its existence and uniqueness. A popular high order model, total generalized variation (TGV), involves high order derivatives and automatically balances the first to \(k\text {th}\) derivatives [13]. The second order TGV is defined as following:

$$\begin{aligned} E(u)_{TGV}=\text {TGV}_{\alpha }^{2} = \alpha _{1} \int _{\varOmega }|\nabla u -v|dxdy + \alpha _{2} \int _{\varOmega }|\varepsilon (v)|dxdy, \end{aligned}$$
(4)

where the minimum is taken over the vector fields v and \(\varepsilon (v)=\frac{1}{2}(\nabla v + \nabla v^{\text {T}})\) denotes the symmetrized derivative. TGV reduces the staircase effect and leads to piecewise polynomial intensities [14]. The connections between some typical additive high order models are detailed in [15]. Typical non-variational methods includes bilateral filter [16], nonlocal means filter [17, 18], guided filter [19] and BM3D [20].

In this paper, we will introduce piecewise harmonic image, which is more smoother beyond the classical piecewise constant image and piecewise linear image. It allow a weak edge between different regions and it is difficult to recovery it. We will present a a new model to address this problem. The rest of this paper is organized as follows. Since our aim is to recover a more smoothing image, Sect. 2 introduce the definition of harmonic image and a new multiplicative framework for model construction. A new high order model is presented and its features are discussed in Sect. 3, Experimental results are shown in Sect. 4 and a brief conclusion is given in Sect. 5.

2 Framework for Piecewise Smoothing Image Recovery

2.1 Piecewise Smoothing Image: From Constant to Harmonic

Let \(\varOmega _{i}\),\(i=1,2,\cdots ,n\), be a partition of \(\varOmega \). A common piecewise image is defined as

$$\begin{aligned} u(x,y)=\sum _{i=0}^{m}u_{i}(x,y), \end{aligned}$$
(5)

where

$$\begin{aligned} u_{i}(x,y)=\left\{ \begin{aligned} \text {smoothing image}&(x,y)\in \varOmega _{i}, \\ 0&\text {otherwise}. \\ \end{aligned} \right. \end{aligned}$$
(6)

We require that the smoothing image in (6) is continuous and differentiable in every partition \(\varOmega _{i}\). An usual way is to use homogeneous polynomial to represent the smoothing function. Therefore, image I is named as a piecewise constant image when \(u(i)=c_{i}\) and a piecewise linear or affine image when \(u(i) = a_{i}x+b_{i}y+c_{i}\). TV can recover piecewise constant image successfully. Several high order models have been proposed to recover piecewise linear image.

Fig. 1.
figure 1

Two harmonic functions and their corresponding images. The left-up is the shape of the harmonic function \(f(x,y) = \frac{x^2}{a^2}-\frac{y^2}{b^2}\) and right-up is its corresponding image. The right-down is the shape of the harmonic function \(f(x,y) = \frac{y}{x}\) and right-up is its corresponding image.

Full size image

In this paper, for the first time, we consider the recovery of a class more smoothing image: piecewise harmonic image. In mathematic, a function f is said to be harmonic if it satisfies the following Laplace equation:

$$\begin{aligned} \triangle f(x,y) = \frac{\partial ^2 f}{\partial x^2} + \frac{\partial ^2 f}{\partial y^2}=0. \end{aligned}$$
(7)

Except the traditional constant function and the linear ones, many more smoothing function are harmonic. A quadratic one is a special case when \(a=b\) for the hyperbolic paraboloid in geometry:

$$\begin{aligned} f(x,y) = \frac{x^2}{a^2}-\frac{y^2}{b^2}, \end{aligned}$$
(8)

which describes the shape for a doubly ruled surface in 3D space. Another example is \(f(x,y) = \text {arctan}(\frac{y}{x})\), which has a non-vanishing derivatives to infinity. Figure 1 illustrates the the profiles of two harmonic functions and their corresponding images. These two functions and images has a complex and smoothing geometrical structure. Therefore, an image u is said to be a piecewise harmonic if u(i) meets Eq. (7) in partition \(\varOmega _{i}\). It permits more wild range smoothing functions beyond polynomial, though it is an extension to the traditional piecewise constant image and linear image. The sharp edges between the piecewise constant are easy to preserved and it is difficult to preserve the edges between the different harmonic regions, as its gradient may be small.

2.2 Multiplicative High Order Variational Framework

Based on the decisions above, we may infer that it is a challenge to recover piecewise harmonic image as it permits more smoothing structures beyond constant region and affine region. Before constructing a feasible variation model to this problem, we should consider two issues. The first is how to judge where is the boundaries of different smoothing transition regions, which is helpful for a reasonable piecewise. The second is how to choose a proper way to describe the smoothing function, which is responsible for smoothing control. The answers to the two problems need to be integrated into the variational model. To improve the smoothing degree of the restored image, one need to incorporate high order operator to describe the smoothing requirement. Therefore, we proposed the following general high order framework for piecewise smoothing image recovery:

$$\begin{aligned} E(u)=\int _{\varOmega }f_{\text {p}}(u,u_{i},u_{ij})f_{\text {s}}(u,u_{i},u_{ij})dxdy, \end{aligned}$$
(9)

where \(f_{\text {p}}\) provides the clues for judging the boundaries between piecewise regions and \(f_{\text {s}}\) conveys the smoothing control respectively. Contrary to the traditional additive high order variational model framework, the multiplicative model is easy to extend to other imaging tasks.

3 Proposed Weighted Laplacian Model

By consider a gray scale image u(x, y) as a surface \(S=(x,y,u(x,y))\), we propose the the following geometrical weighted Laplace (GWL) energy functional:

$$\begin{aligned} E_{\text {GWL}}(u) =\int _{\varOmega } \frac{|\triangle u|}{\sqrt{1+|\nabla u|^2}}dxdy. \end{aligned}$$
(10)

The kernel in the the energy (10) is a product of two functions and it can be seen as a special case for (9) when choosing \(f_{\text {p}} = \frac{1}{\sqrt{1+|\nabla u|^2}} \) and \(f_{\text {s}}= |u_{xx}+u_{yy}| = |\triangle u|\).

The key of recovery of the piecewise harmonic image is the interaction between two functions.

  1. 1.

    Piecewise. The piecewise effect in a certain partition is guaranteed by edge boundary detector \(g= \sqrt{1+|\nabla u|^2}\), which has a remarkable geometrical interpretation:

    $$\begin{aligned} r = \frac{1}{g}=\frac{1}{\sqrt{1+u_{x}^2+u_{y}^2}}=\frac{dxdy}{gdxdy}= \frac{A^{\text {domain}}}{A^{\text {surface}}}, \end{aligned}$$
    (11)

    where \(A^{\text {domain}}\) is the area of the infinitesimal surface in the image domain (x, y), and \(A^{\text {surface}}\) is its corresponding area on the image surface (x, y, u(x, y)). Therefore, r conveys the height variation on the surface as well the intensigy variation on the image data [21]. r is equal 1 for flat surface and its Laplacian is zero too, such structure will be preserved. r is equal 0 near edges, which is helpful to preserve edges.

  2. 2.

    Harmonic. The smoothing harmonic constrain is mainly performed by Laplacian operator \(\varDelta u\). As \(g=\sqrt{1+|\nabla u|^2}>1\), zero Laplacian means the kernel function will be zero too and functional reaches the minimizer in this region. Therefore, smoothing structures will be kept if they can be represented as any harmonic function.

  3. 3.

    Edge preserving. For an ideal typical sharp edge, its Laplace has a famous zero crossing property: near the midpoint of the edge, its second order derivative would cross zero. The kernel function will be 0 as \(\varDelta u = 0 \) and \(r=0\) for a true sharp edge, which will be recognized and well preserved.

Therefore, the proposed model permits discontinuous while preserving piecewise smoothing regions.

Adding an artificial time to the Euler-Lagrange equation derived to (10), we can obtain the following an anisotropic high order nonlinear diffusion equation:

$$\begin{aligned} u_{t}= - \triangle \left( \frac{\triangle u}{g|\triangle u|} \right) + \mathrm{div} \left( \frac{|\triangle u|}{g^3}\nabla u\right) -\lambda (u-u_{0}). \end{aligned}$$
(12)

The initial condition is \(u(x,0)=u_{0}\) and its boundary condition is

$$\begin{aligned} (u_x,u_y)\cdot \mathbf {\mu }=0, (\gamma _{1},\gamma _{2})\cdot \mathbf {\mu }=0, \end{aligned}$$
(13)

where \(\mathbf {\mu }\) is the unit outward normal direction to \(\partial \varOmega \) and \(\gamma _{1}\) and \(\gamma _{2}\) are defined as

$$\begin{aligned} \gamma _{1}=\left( \frac{\triangle u}{g|\triangle u|}\right) _{x}+ \frac{|\triangle u|u_{x}}{g^{3}},\end{aligned}$$
(14)
$$\begin{aligned} \gamma _{2}=\left( \frac{\triangle u}{g|\triangle u|}\right) _{y}+ \frac{|\triangle u|u_{y}}{g^{3}}. \end{aligned}$$
(15)

The diffusion of Eq. (12) is decided by the interaction of the first order edge detector g and the second order information \(\frac{\triangle u}{|\triangle u|}\). Noting \(\frac{\triangle u}{|\triangle u|} = \mathrm{sign}\triangle u\), only three values, \(-1, 0, 1\) are permitted. When it equals 0, the diffusion stop automatically. It means that the local structure described by the harmonic function maybe preserved. When it equals 1 or \(-1\), the diffusion now depends on the magnitude the boundary detector. The diffusion speed will slow down as the sign of the Laplace operator is scaled by the inverse of a large gradient magnitude. A fast diffusion will be performed in flat region as the image gradient is small and the boundary detector \(g \simeq 1\).

As the evolution equation is nonlinear highly, we now consider to solve it by an explicit finite difference method. For time discretion, forward difference is used and the space grid size is set as \(h=1\). Table 1 lists the scheme for time and spatial operators in high order nonlinear Eq. (12).

Table 1. The discrete scheme for operators in high order nonlinear Eq. (12).
Full size table

4 Experimental Results

In this section, we conduct several experiments to demonstrate the performance of the high order GWL model. We make comparisons with three related methods. The first one is second order TV method, which is famous for its edge preserving ability. The second method is TGV method, which is implemented by a primal-dual splitting method in [22]. The code is also available: http://www.gipsa-lab.fr/~laurent.condat/software.html. The third one is state of art BM3D method. To do a quantitative comparison, peak signal-to-noise-ratio (PSNR) is used for quantitative comparison. For the proposed method, we set time space \(\triangle t = 10^{-2}\) and \(\lambda = 0.01\).

Fig. 2.
figure 2

Piecewise quadratic denoised images. The image is composed by two constant functions (one for left side and another for right side), a linea function (up middle and down middle) and a selected quadratic harmonic function for \(u(x,y) = \frac{x^2}{16}-\frac{y^2}{16}\). From the left to right, the first row: clean image, noisy image (PSNR = 28.1376), TV result (PSNR = 43.3636). From the left to right, the second row, from the left to right, TGV result(PSNR = 32.3433), BM3D result (PSNR = 47.0606), GWL result (PSNR = 49.3684).

Full size image
Fig. 3.
figure 3

The induced surfaces of piecewise quadratic denoised images. The order is the same as Fig. 2.

Full size image
Fig. 4.
figure 4

Piecewise smoothing denoised images beyond quadratic. It is composed of a linear function and a smoothing function \(u(x,y) = \text {arctan}{(\frac{y}{x}})\), whose infinite derivatives are non-vanishing. From the left to right, the first row: clean image, noisy image (PSNR = 28.1221), TV result (PSNR = 43.2176). From the left to right, the second row, from the left to right, TGV result (PSNR = 31.8597), BM3D result (PSNR = 45.4445), GWL result (PSNR = 49.6065).

Full size image
Fig. 5.
figure 5

The induced surfaces of piecewise smoothing denoised images in Fig. 4.

Full size image
Fig. 6.
figure 6

Nasa denoised image. From the left to right, the first row: clean image, noisy image (PSNR = 22.1151), TV result (PSNR = 40.1939). From the left to right, the second row, from the left to right, TGV result (PSNR = 33.5203), BM3D result, GWL result (PSNR = 42.2764).

Full size image

The first experimental results on a synthesized piecewise quadratic image are shown in Fig. 2. The test image is composed by two constant functions (one for left side and another for right side), a linea function (up middle and down middle) and a selected quadratic harmonic function for \(u(x,y)=\frac{x^2}{16}-\frac{y^2}{16}\) in (8) (middle). The noise level is 10 and the denosing results for the noisy image (PSNR = 28.1376) by four methods are shown in 2. The staircase effect is obvious in quadratic region for TV denoised image (PSNR = 43.3636). The TGV denoised image (PSNR = 32.3433) shows a good smoothing ability but blurs the edges seriously. The staircase effect in linear regions and quadratic regions is unpleasant in visual for BM3D denoised image (PSNR = 47.0606). The proposed GWL method provide an almost perfect denoised image visually and quantitatively (PSNR = 49.3684). The corresponding induced surfaces are displayed in Fig. 3. It can be observed that TGV and GWL shows a better smoothing effect than TV and BM3D.

The second test synthesized image has a more complex structures: it is composed of a linear function and a smoothing function \(u(x,y) = \text {arctan}{\frac{y}{x}}\), whose infinite derivatives are non-vanishing. The noise level is 10 and the denosing results for the noisy image (PSNR = 28.1221) by four methods are shown in 4. TV result shows a serious staircase effect for the tangent function region (PSNR = 43.2176). The TGV denoised image (PSNR = 31.8597) blurs the edges heavily again. BM3D performs better than TV and TGV but staircase effect is visual for linear region (PSNR = 45.4445). The proposed GWL method yields a best result among four methods visually and quantitatively (PSNR = 49.6065). The corresponding induced surfaces are displayed in Fig. 5.

The third test image is a picture of moon rise captured from the space station by NASA astronaut Randy Bresnik on August 3, 2017. The noise level is 20 and the denosing results for the noisy image (PSNR = 22.1151) by four methods are shown in Fig. 6. Four methods remove noise in white and black background. The differences between them lie in the moon surface and the smoothing transition regions in the middle of the image. BM3D provides the best detail preservation ability for moon surface (PSNR = 41.3395) while GWL produces a good transition effect between the white region and black region (PSNR = 42.2764). TV still suffers from the staircase (PSNR = 40.1939) and TGV blurs edges (PSNR = 33.5203).

5 Conclusions

We present a high order variational method to recover a class more smoothing piecewise image beyond quadratic, which we call piecewise harmonic image. Piecewise harmonic image covers the popular piecewise constant and piecewise linear images and beyond them, even including some certain function with infinite order non-vanishing derivatives. We construct the new model within a multiplicative variational framework and its kernel is based is based on a geometrical weighted Laplacian operation. The research in this paper shows that we can restore piecewise harmonic image perfectly. Its major limitation is the fact that the natural image do not always contain standard piecewise quadratic geometrical structures. Therefor, improvement on its adaptability to more image is part of our future work. Another important work is to devise an efficient speeding up algorithms for GWL model.