Spatial and Frequency-Based Variational Methods for Perceptually Inspired Color and Contrast Enhancement of Digital Images

Abstract

In the past 20 years, variational principles in image processing and computer vision flourished, often allowing a deep comprehension and a more efficient solution to many problems. As this chapter presents, this also holds true for color image processing. We start by discussing the fundamental result about the interpretation of histogram equalization as the minimization of a suitable functional. This functional is given by two opponent terms: the first being a global quadratic adjustment to the middle gray level, and the second representing a global contrast enhancement term. It is proven here that this analytical form is shared by the so-called perceptual functionals, which allow an enhancement of color images in line with the most important human visual system features. For perceptual functionals, the adjustment is entropic and contrast enhancement is local and invariant under global illumination changes. It is also highlighted that the variational setting provides a unified framework where existing perceptually inspired color improvement models can be seen as particular instances. In the end, the wavelet version of this framework is discussed in detail.

Appendices

Appendix 1

We provide here the proof of Theorem 5.1. We think that this proof is instructive because it explicitly shows the link between the variations of the functionals appearing in Theorem 5.1 and histogram equalization, which is far from being intuitive at the first sight.

Let us start with a useful lemma.

Lemma 5.1.

Given the two functionals

$$ E_1(I)=\int_{\Omega} \psi(I(x))\,dx, \quad E_2(I)=\iint_{\Omega^2} \phi(I(x),I(y))\,dxdy, $$

(5.26)

where $\psi$ is a differentiable function defined on the codomain of $I$, and $\phi$ is a differentiable function defined on the 2-th Cartesian power of the codomain of $I$, then their first variations are, respectively:

$$\delta E_1(I,J)=\int_{\Omega} \left.\frac{\partial \psi}{\partial I}\right|_{I(x)} \, J(x) dx \equiv \int_{\Omega} \psi'(I(x)) \, J(x) \, dx$$

(5.27)

and

$$\delta E_2(I,J)=\iint_{\Omega^2} \left(\left.\frac{\partial \phi}{\partial I}\right|_{I(x)} J(x) + \left.\frac{\partial \phi}{\partial I}\right|_{I(y)} J(y) \right) \, dxdy.$$

(5.28)

The proof of this lemma can be found in any book on variational principles and we do not report it here, preferring passing directly to the proof of Theorem 5.1.

Proof

By linearity, we can compute the first variation of the two terms of the energy functional separately and then add the results. For that, let us recall that we have written:

$$ D_{\frac{1}{2}}(I) = 2\int_\Omega \left(I(x)-\frac{1}{2}\right)^2 \, dx;$$

(5.29)

$$ C(I) = \frac{1}{|\Omega|}\iint_{\Omega^2} |I(x)-I(y)| \, dx dy.$$

(5.30)

By virtue of formula (5.27), we have

$$\delta D_{\frac{1}{2}}(I,J)=\int_\Omega 4\left(I(x)-\frac{1}{2}\right) J(x)\, dx,$$

(5.31)

and by virtue of formula (5.28), we have

$$ \begin{aligned} \delta C(I,J) & = \frac{1}{|\Omega|}\iint_{\Omega^2} \left[{\rm sign}(I(x)-I(y))J(x)-{\rm sign}(I(x)-I(y))J(y)\right] \, dx dy \\ & = \frac{1}{|\Omega|}\iint_{\Omega^2} {\rm sign}(I(x)-I(y))J(x) \, dx dy \; + \\ & - \frac{1}{|\Omega|}\iint_{\Omega^2} {\rm sign}(I(x)-I(y))J(y) \, dx dy. \end{aligned} $$

(5.32)

Now, interchanging the role of the “mute” variables $x$ and $y$ in the second integral of the last step, we have that

$$\frac{1}{|\Omega|}\iint_{\Omega^2} {\rm sign}(I(x)-I(y))J(y) \, dx dy=\frac{1}{|\Omega|}\iint_{\Omega^2} {\rm sign}(I(y)-I(x))J(x) \, dy dx$$

(5.33)

but then, using the oddness of the sign function,

$$\frac{1}{|\Omega|}\iint_{\Omega^2} {\rm sign}(I(x)-I(y))J(y) \, dx dy=-\frac{1}{|\Omega|}\iint_{\Omega^2} {\rm sign}(I(x)-I(y))J(x) \, dy dx.$$

(5.34)

Hence, we can write

$$ \begin{aligned} \delta C(I,J) & = \frac{1}{|\Omega|}\iint_{\Omega^2} {\rm sign}(I(x)-I(y))J(x) \, dy dx \; + \\ & + \frac{1}{|\Omega|}\iint_{\Omega^2} {\rm sign}(I(x)-I(y))J(x) \, dy dx = \\ & = \frac{2}{|\Omega|}\iint_{\Omega^2} {\rm sign}(I(x)-I(y))J(x) \, dy dx \end{aligned} $$

(5.35)

that can be conveniently rearranged as follows:

$$\delta C(I,J) = \int_{\Omega} \left(\frac{2}{|\Omega|}\int_\Omega {\rm sign}(I(x)-I(y))\, dy \right) J(x) \, dx.$$

(5.36)

Now, since $\delta E_{\rm hist \; eq}(I,J)=\delta D_{\frac {1}{2}}(I,J)-\delta C(I,J)$, by using formulas (5.31) and (5.36) we have

$$\delta E_{\rm hist \; eq}(I,J)= \int_\Omega 4\left(I(x)-\frac{1}{2}\right) J(x)\, dx - \int_{\Omega} \left(\frac{2}{|\Omega|}\int_\Omega {\rm sign}(I(x)-I(y))\, dy \right) J(x) \, dx$$

(5.37)

that is,

$$\begin{aligned} \delta E_{\text{hist eq}}(I,J)= \int_\Omega \left[ 4\left( I(x)-\frac{1}{2}\right) - \frac{2}{|\Omega|}\int_\Omega \text{sign}(I(x)-I(y))\, dy \right] J(x)\, dx.\end{aligned}$$

(5.38)

The stationary condition $\delta E_{\rm hist \; eq}(I,J)=0$, $\forall J$, implies that the expression in the square bracket must be zero, that is,

$$ \delta E_{\rm hist \; eq}(I,J)=0 \quad \Longleftrightarrow \quad 4\left(I(x)-\frac{1}{2}\right) - \frac{2}{|\Omega|}\int_\Omega {\rm sign}(I(x)-I(y))\, dy=0,$$

(5.39)

so that the Euler–Lagrange equation relative to the energy functional $E_{\rm hist \; eq}$ is the following implicit equation

$$2\left(I(x)-\frac{1}{2}\right) - \frac{1}{|\Omega|}\int_\Omega {\rm sign}(I(x)-I(y))\, dy=0,$$

(5.40)

that can be suitably rewritten as

$$\frac{1}{|\Omega|}\int_\Omega {\rm sign}(I(x)-I(y))\, dy=2I(x)-1.$$

(5.41)

Now, using the identity ${\rm sign} (t)=2{\rm sign} ^+(t)-1$, we can express the left-hand side of the Euler–Lagrange equation as

$$ \begin{aligned} \frac{1}{|\Omega|}\int_\Omega (2{\rm sign}^+(I(x)-I(y))-1)\, dy & =\frac{2}{|\Omega|}\int_\Omega {\rm sign}^+(I(x)-I(y))\, dy - \frac{\int_\Omega \, dy}{|\Omega|} \\ & = 2H(I(x))-1, \end{aligned} $$

(5.42)

where we have used the fact that $\frac {1}{|\Omega |}\int _\Omega {\rm sign} ^+(I(x)-I(y))\, dy$ is the spatial version of the cumulative histogram $H(I(x))$, as noticed in Eq. (5.5).

Thus, the Euler–Lagrange Eq. (5.41) is equivalent to $2H(I(x))-1 = 2I(x)-1$, that is, to $H(I(x))=I(x)$, but then

$$\delta E_{\rm hist \; eq}(I,J)=0 \quad \Longleftrightarrow \quad H(I(x))=I(x), \; \forall x\in \Omega,$$

(5.43)

which means that the image function $I$ which satisfies the Euler–Lagrange equations of the functional $E_{\rm hist \; eq}(I)$ has an equalized histogram.

The proof of existence and uniqueness of the solution of the gradient descent scheme written in Theorem 5.1 is quite long and technical and can be foundin[22]. □

Appendix 2

Let us start this section by briefly recalling the basic information about wavelet theory in 1D, then we will extend the discussion to 2D wavelets, the main reference for all the results quoted hereafter is [14]. A 1D (mother) wavelet $\psi \in L^2(\mathbb{R} )$ is a unit norm and null-mean function. Of course this is possible only if $\psi$ oscillates, but, unlike infinite waves, wavelets can have compact support. The $\psi$-wavelet transform $W_\psi f$ of $f\in L^2(\mathbb{R} )$ in the point $\xi$ at the scale $s$ is given by the inner product $W_\psi f(\xi ,s) = \int _\mathbb {R} f(x) \frac {1}{\sqrt {s}} \overline \psi \left ( \frac {x-\xi }{s} \right ) dx$. $W_\psi f$ gives a “measure of similarity” between $f$ and $\psi$ around the point $\xi$ at the scale $s$. So, if a signal is constant or do not vary “too much” in the support of a wavelet, then its wavelet transform will be zero or very small, this is how wavelets provide a multiscale information about the local contrast of a signal.

The set $\{\psi _{j,k}\}_{(j,k)\in \mathbb {Z}^2} \subset L^2(\mathbb{R} )$ given by $\psi _{j,k}(x)\equiv \frac {1}{\sqrt {2^j}}\psi \left (\frac {x-2^j k}{2^j}\right )$ is a complete orthonormal system of $L^2(\mathbb{R} )$. Moreover, $L^2(\mathbb{R} )$ can be recovered by the closure of the union of a sequence of nested closed subspaces $V_j \subset V_{j-1}$ with suitable properties (see Mallat’s book [14] for more details). The orthogonal projections of $f\in L^2(\mathbb{R} )$ onto $V_j$ and $V_{j-1}$ give the approximation of $f$ at the scales $2^j$ and $2^{j-1}$, respectively. The $2^j$-approximation is coarser and the missing details with respect to the finer $2^{j-1}$-approximation are contained in the orthogonal complement $W_j$ of $V_j$ in $V_{j-1}$: $V_{j-1} = V_j \oplus W_j$. $W_j$ is called the $j$-th detail space and it can be proven that the orthogonal projection of $f$ on $W_j$ is given by $P_{W_j} f = \sum _{k\in \mathbb Z} \langle f,\psi _{j,k} \rangle \psi _{j,k} \equiv \sum _{k\in \mathbb Z} d_{j,k} \psi _{j,k}$. The coefficients $d_{j,k}$ are called detail coefficients of $f$ at the scale $2^j$. Fine-scales detail coefficients at fine scale are sparse, in fact, they are non-null only when the support of $\psi _{j,k}$ intersects a high contrast zone, that is, around sharp edges.

Finally, let us recall that every wavelet $\psi$ is related to a mirror filter $h$ and to a function $\phi$, called scale function, through the following equation that involves their Fourier transforms: $\hat \psi (2\omega ) = \frac {1}{\sqrt {2}} e^{-i\omega } \hat h^\ast \left ( \omega + \pi \right ) \hat \phi \left ( \omega \right )$, see [24] for a complete and detailed description of how to generate wavelets using the filter design methodology. $\phi$ appears in the orthogonal projection of a signal $f$ onto the approximation space $V_j$, in fact it can be proven that $P_{V_j} f = \sum _{k\in \mathbb Z} \langle f,\phi _{j,k} \rangle \phi _{j,k} \equiv \sum _{k\in \mathbb Z} a_{j,k} \phi _{j,k}$, where $\phi _{j,k}(x) = \frac {1}{\sqrt {2^j}}\phi \left (\frac {x-2^j k}{2^j}\right )$ and $a_{j,k}$ are called approximation coefficients at the scale $2^j$. It follows that $P_{V_{j-1}} f = P_{V_j}f + P_{W_j} f = \sum _{k\in \mathbb Z} a_{j,k} \phi _{j,k} + \sum _{k\in \mathbb Z} d_{j,k} \psi _{j,k}$.

In practical applications one is interested in a multiresolution analysis between two fixed scales $2^L$ and $2^J$, $L,J\in \mathbb{Z}$, $L<J$. In this case $V_{J-1} = V_J \oplus W_J$, $V_{J-2} = V_{J-1} \oplus W_{J-1} = V_J \oplus W_J \oplus W_{J-1}$ and so on, thus $V_{L} = V_{J} \oplus \bigoplus _{2^J \geq 2^j \geq 2^{L+1}} \; W_j$. For this reason, following [14], we say that a discrete orthogonal wavelet multiresolution representation of a 1D signal $f$ between two fixed scales $2^L$ and $2^J$, $L,J\in \mathbb{Z}$, $L<J$, is given by the collection of detail coefficients $\{d_{j,k}\}$ at all scales, completed by the approximation coefficients at the coarser scale, that is, $\{a_{J,k}\}$.

When we deal with 2D signals, as images, we have to consider a multiresolution analysis of $L^2(\mathbb{R} ^2)$. Multidimensional wavelet bases can be generated with tensor products of separable basis functions defined along each dimension. In this case, an orthogonal wavelet multiresolution representation between two scales $2^L$ and $2^J$, $L,J\in \mathbb{Z}$, $L<J$, is given by three sets of detail coefficients $\{d^H_{j,k},d^V_{j,k},d^D_{j,k}\}$ at all scales, which correspond to the horizontal, vertical, and diagonal detail coefficients, respectively, completed by the approximation coefficients at the coarser scale, that is, $\{a_{J,k}\}$.