A Gradient-Based Spatial Color Algorithm for Image Contrast Enhancement

Abstract

This work presents GREAT-Mix, a novel contrast enhancer derived from the combination of the Retinex-inspired spatial color algorithms STRESS and GREAT. These algorithms improves the visibility of the details and content of any input image by adjusting its colors on the basis of local spatial and visual features processed channel by channel. Precisely, STRESS stretches the intensity I(x) of each pixel x between two bounding values corresponding to the minimum and maximum intensities of sets of pixels randomly sampled with radial density around x, while GREAT re-scales I(x) by a factor obtained by processing a set of edges deterministically selected over the image. GREAT-Mix implements the stretching function of STRESS but determines its bounding values from the edges sampled by GREAT. The result is a new spatial color algorithm that performs contrast enhancement similarly to STRESS, but that, thanks to the deterministic sampling of GREAT, grants robustness to noise and repeatability of the outcomes.

1 Introduction

A contrast enhancer is an algorithm that takes as input an image and increases the visibility of its details, making the image content more understandable for human observers while highlights visual cues relevant to machine vision tasks.

Among the many contrast enhancement techniques proposed in the literature (e.g. [2, 3, 5, 22]), here we focus on the algorithms GREAT [15] and STRESS [6], that belong to the Milano Retinex (Mi-Retinex for short) family [24]. This latter is a set of spatial color algorithms grounded on the famous Retinex theory [8] and widely employed to enhance real-world images. Mi-Retinexes are of interest because the enhancement techniques they propose implement two important characteristics of the human color vision system, which are also at the basis of Retinex: (i) the independent analysis of the color components of the visual signal; (ii) the color adjustment based on local spatial and visual information. According to (i) and (ii), Mi-Retinexes process the R, G, B channel of any input image separately and they map the intensity I(x) of each pixel x of the channel $I = R, G, B$ to a new value obtained by processing the visual features extracted from a set of pixels sampled around x. The result is a new image, named lightness, which is an enhanced version of the input one: the lightness is usually brighter and more contrasted than the input one, while possible shadows and dominant cast due to the light are lowered. In case of real-world images, the Mi-Retinex lightness at any pixel x is usually obtained by dividing the channel intensity I(x) by a strictly positive value $w_+(x) \ge I(x)$ called local reference white (LRW). Mi-Retinexes propose different spatial sampling procedures and equations for the LRW. For instance, the spatial sampling is performed by exploring the region around each pixel by Brownian paths in [19], by random paths proximate to edges in [13, 20, 25, 26], by sets of unconnected pixels [16] chosen randomly in [12, 14, 21] while deterministically in [9, 11, 15]. Such paths and sets are the geometric support from which to extract the features that contribute to the LRW equation and that may include, in addition to the intensity, gradient and/or spatial information as e.g. in [9, 11,12,13,14,15, 25, 26]. The different sampling schemes and the different equations of LRW yield to different enhancement levels and, in general, the use of one algorithm instead of another is driven by the tasks at the hand.

In this work, we present GREAT-Mix, a novel contrast enhancer derived from the combination of the Mi-Retinex algorithm GREAT [15] and the Mi-Retinex inspired algorithm STRESS [6]. GREAT proposes an interesting spatial sampling scheme that cuts down the complexity of the pixel-wise sampling process of many previous Mi-Retinexes. GREAT selects from each image channel I a set of edges with high gradient magnitude and uses them to compute pixel by pixel the LRW. The LRW at a pixel x is the average of the intensities exceeding I(x) and belonging to the set of selected edges. To model the locality of the color adjustment and to account for the edge strength, in the LRW equation, each intensity is weighted by a term inversely proportional to the distance of the corresponding edge from x and directly proportional to its gradient magnitude. STRESS replaces the Mi-Retinex scaling function with a new operation: this stretches the intensity I(x) between two bounding values computed by re-working the minimum and maximum intensities of random sets of pixels radially distributed around x [21]. The main disadvantage in using STRESS is due to the chromatic noise that random sampling may introduce in the enhanced image and that is particularly evident when few sprays are considered. The re-formulation of STRESS in to a population based model proposed in [4] (STRESS-P) avoids the random sampling and thus yields a noise-free lightness but at the prize of a much higher computational complexity and thus of a longer execution time.

The algorithm GREAT-Mix combines the sampling strategy of GREAT with the stretching function of STRESS, leading to a new contrast enhancer with a performance similar to that of STRESS but with the advantages of no chromatic noise generation, result repeatability and computational efficiency inherited from GREAT. The name ‘GREAT-Mix’ assembles the word ‘GREAT’ and the contraction of the words ‘MInimum, maXimum’ that refer to the bounding values of STRESS and at the same time it reminds that GREAT-Mix is obtained by ‘mixing’ the elements of other algorithms.

The paper outlines as follows: Sect. 2 describes GREAT, STRESS and STRESS-P; Sect. 3 explains GREAT-Mix; Sect. 4 reports the performance analysis of GREAT-Mix, while Sect. 5 draws conclusions and future work.

2 Related Works: GREAT and STRESS

The following notation will be used throughout the paper. Let ${\overline{I}}$, I and L be respectively a color image, a channel of ${\overline{I}}$ and the lightness of I. Here, I and L are represented as the functions $I, L: S \rightarrow (0, 1]$, where S is the support of I, i.e. the set of spatial coordinates of the pixels of I, and the intensity values of I and L are re-scaled to (0, 1], with zero excluded to prevent division by zero.

2.1 GREAT

GREAT (from Gradient RElevAnce for ReTinex) enhances each image channel I by two steps:

• 1. GREAT Global Processing: GREAT extracts from I the set $\mathcal {R}$ of edges whose gradient magnitude exceed a pre-defined threshold $\tau \le 1.0$, i.e.:

  $$\begin{aligned} \mathcal {R} = \{y \in S: \; \parallel \nabla I(y) \parallel \ge \tau \}, \end{aligned}$$

  (1)

  where $\nabla I$ indicates the gradient magnitude of I that here is normalized to range over [0, 1].

  Then, GREAT defines a function $M_+: S \rightarrow (0, 1]$ that assigns to each pixel $y \in \mathcal {R}$ the maximum intensity value in a $3 \times 3$ window N(y) centered at y, while sets to zero the values of $S \setminus \mathcal {R}$, i.e.:

  $$\begin{aligned} {\mathcal {M}}_+(y) = \left\{ \begin{array}{ll} \max \{I(u): u \in N(y)\} &{} \; \; \forall \; y \in \mathcal {R}\\ 0 &{} \; \; \mathrm{otherwise} \\ \end{array} \right. \end{aligned}$$

  (2)

• 2. GREAT Pixel-wise Processing: for each $x \in S$, GREAT computes:

  • the set $P_+(x)$ = $\{ u \in S: {\mathcal {M}}_+(u) > I(x)\}$.

  • the value $w_+(x) \in (0, +\infty )$ given by:

    $$\begin{aligned} w_+(x) = \left\{ \begin{array}{ll} \frac{\sum _{u \in P_+(x)} (1 - d(u, x)) \parallel \nabla I (u) \parallel I(u)}{\sum _{u \in P_+(x)} (1 - d(u, x)) \parallel \nabla I (u) \parallel } &{} \; \; \mathrm{if } P_+(x) \ne \emptyset \\ I(x) &{} \; \; \mathrm{otherwise} \end{array} \right. \end{aligned}$$

    (3)

    where d(u, x) indicates the Euclidean distance between u and x, divided by the length of the diagonal of the image support.

  • The value L(x) is given by the ratio $L_\mathrm{GREAT}(x) = \frac{I(x)}{w_+(x)}$.

The choice of strong edges (i.e. pixels with high gradient magnitude) as features relevant to the LRW is justified by the importance that strong contrasts play in human color vision [7]. The threshold $\tau$ determines the cardinality of $\mathcal {R}$, and thus the complexity of GREAT, which is $\mathcal {O}(|\mathcal {R}||I|)$. The work in [15] suggests to set $\tau$ unsupervisely as the mean value of the image gradient magnitude, i.e.:

$$\begin{aligned} \tau = \frac{1}{|S|} \sum _{y \in S} \parallel \nabla I(y) \parallel \end{aligned}$$

(4)

An executable file of GREAT, running on Windows $10 \times 64$, is available at [10].

2.2 STRESS

For each channel I, STRESS (from Spatio-Temporal Retinex-Inspired Envelopes with Stochastic Sampling) defines two functions $E_m, E_M: S \rightarrow \mathbf {R}$ bounding I, i.e. for every $x \in S$, $E_m(x) \le I(x) \le E_M(x)$. The functions $E_m$ and $E_M$ are called respectively the minimum and maximum envelopes of I and they are computed pixel by pixel as follows. The neighborhood N(x) of each pixel x. is sampled by n random sprays $S_1(x), \ldots , S_n(x)$, where, for any $k = 1, \ldots , K$, $S_k(x)$ is a set of p pixels randomly selected with radial density around x (see Fig. 1(b) and the work [21] where random sprays have been first introduced). The values $E_{m}(x)$ and $E_{M}(x)$ are computed by the following equations:

$$\begin{aligned} E_{m}(x) = I(x) - R(x)v(x), \; \; E_{M}(x) = E_m(x) + R(x) \end{aligned}$$

where $R(x) = \frac{1}{N}\sum _{k=1}^{n}R_{k}(x)$ and $v(x) = \frac{1}{N}\sum _{k=1}^{n}v_{k}(x)$, with

$$\begin{aligned} R_k(x) = E_{\max }^k(x) - E_{\min }^k(x) , \; \; v_k(x) = \left\{ \begin{array}{ll} \frac{1}{2} &{} \; \mathrm{if }\; R_k(x) = 0\\ \frac{I(x)-E_{min}^{k}(x)}{R_{k}(x)} &{} \; \mathrm{otherwise} \end{array} \right. \end{aligned}$$

and $E_{\min }^{k}(x)$ and $E_{\max }^{k}(x)$ are respectively the minimum and maximum intensities of the spray $S_k(x)$:

$$\begin{aligned} E_{\min }^{k}(x) = \min \left\{ I(y):y\in S_{k}(x)\right\} , \; \; E_{\max }^{k}(x) = \max \left\{ I(y):y\in S_{k}(x)\right\} \end{aligned}$$

(5)

Finally, the lightness at x is given by:

$$\begin{aligned} L_\mathrm{STRESS}(x) = \left\{ \begin{array}{ll} \frac{1}{2} &{} \mathrm{if } \; E_{M}(x) = E_{m}(x)\\ \frac{I(x)-E_{m}(x)}{E_{M}(x)-E_{m}(x)} &{} \mathrm{otherwise } \end{array} \right. \end{aligned}$$

Figure 2(a) shows an example of envelopes computed on the red channel of the color image displayed in Fig. 1(a, left) along its version enhanced by STRESS (a, middle).

The computational complexity of STRESS is $\mathcal {O}(np)$, where p tunes the locality of the spatial processing, and n controls the level of noise due to the random sampling. This noise is the more evident the lower the number of sprays is. The values of p abd n are input user, generally fixed by a trial-and-error procedure as a compromise between enhancement level and image quality. Noise generation is completely avoided by STRESS-P [4], which is the exact mapping of STRESS into a population based model. Basically, STRESS-P estimates the lower and upper bounds of I(x) as the probability to pick up the minimum and the maximum intensities from p points sampled from the probability density function pdf(x) of the variable $I_x: S \setminus \{x\} \rightarrow (0, 1]$ such that for any $y \in S \setminus \{x\}$, $I_y(x) = I(y) D^2 / \parallel x - y \parallel ^2$ and D is the diagonal of S. It is to note that $I_x$ is a spatially weighted version of the intensity I.

While STRESS-P grants robustness to chromatic noise, it requires long computational time, mainly due to the pixel-wise computation of the function pdf (see Sect. 4). The computational complexity of STRESS-P is $\mathcal {O}(|I|^2)$, where |I| is the cardinality of S.

Fig. 1.

From left to right: (a) an image from SCA-30 and its versions enhanced by STRESS and GREAT-Mix; and pixels sampled by (b) STRESS around the barycenter of the support and by (c) GREAT on the red, green and blue image channels. (Color figure online)

Fig. 2.

Minimum and maximum envelopes of the red channel of Fig. 1 by (a) STRESS and (b) GREAT-Mix. (Color figure online)

3 GREAT-Mix

GREAT-Mix adopts the 2-step computational scheme of GREAT, while replaces its intensity re-scaling with the stretching operation of STRESS. Precisely:

• 1. GREAT-Mix Global Processing: GREAT-Mix computes the set $\mathcal {R}$ in Eq. (1) and the map $\mathcal {M}_+$ in Eq. (2). Additionally it computes the map $\mathcal {M}_-: S \rightarrow (0, 1]$ such that

  $$\begin{aligned} {\mathcal {M}}_-(y) = \left\{ \begin{array}{ll} \min \{I(u): u \in N(y)\} &{} \; \; \forall \; y \in \mathcal {R}\\ 0 &{} \; \; \mathrm{otherwise} \\ \end{array} \right. \end{aligned}$$

  (6)

  where N(y) is defined as in GREAT.

• 2. GREAT-Mix Pixel-wise Processing: GREAT-Mix computes

  • the set $P_+(x)$ and the value $w_+(x)$ as defined in the pixel-wise processing of GREAT;

  • the set $P_-(x) = \{u \in S: \mathcal {M}_-(x) \le I(x)\}$;

  • the value $w_-(x) \in (0, +\infty )$ such that

    $$\begin{aligned} w_{\mathcal {M}-}(x) = \left\{ \begin{array}{ll} \frac{\sum _{u \in P_-(x)} (1 - d(u, x)) \parallel \nabla I (u) \parallel I(u)}{\sum _{u \in P_-(x)} (1 - d(u, x)) \parallel \nabla I (u) \parallel } &{} \; \; \mathrm{if } P_-(x) \ne \emptyset \\ I(x) &{} \; \; \mathrm{otherwise} \end{array} \right. \end{aligned}$$

    (7)

    where d(u, x) is the normalized Euclidean distance defined in GREAT.

  • The value L(x) output by GREAT-Mix is given by:

    $$\begin{aligned} L_\mathrm{GREAT-Mix}(x) = \left\{ \begin{array}{ll} \frac{I(x) - w_-(x)}{w_+(x) - w_-(x)} &{} \; \mathrm{if}\; w_+(x) \ne w_-(x) \\ 1 &{} \; \mathrm{otherwise} \end{array} \right. \end{aligned}$$

    (8)

The functions $w_+, w_-: S \rightarrow (0, 1]$ are the envelopes of GREAT-Mix: an example of $w_+$ and $w_-$ is shown in Fig. 2(b) and refers to the red channel of the image in Fig. 1(a, left), whose GREAT enhancement is displayed in Fig. 1(a, right).

4 Experiments

The performance of GREAT-Mix has been evaluated on the public dataset SCA-30 [1]. SCA-30 includes 30 real-world images of indoor and outdoor environments captured with different devices and under different light conditions that make the images challenging to test spatial color algorithms. Some images of SCA-30 have been also employed to evaluate previous Mi-Retinexes, e.g. [9, 11, 16].

The evaluation reported here rely on the analysis of the following five features that are usually modified by enhancement:

• 1. Mean brightness B: the brightness $\mathcal {B}$ of a color image ${\overline{I}}$ is the gray-level image obtained by averaging pixel by pixel the color components of ${\overline{I}}$. B is the mean value of the intensities of $\mathcal {B}$. In general, an image with good detail visibility has a not too low nor too high value of B. In fact, very low (high, resp.) values of B correspond to dark (saturated, resp.) images, whose content is usually poorly understandable (lost, resp.);

• 2. Multi-resolution Contrast C [23] of $\mathcal {B}$: according to [23], the contrast c(x) at a pixel x of $\mathcal {B}$ is the mean value of the $L^1$ distances between $\mathcal {B}(x)$ and the intensities values in a $3\times 3$ window centered at x, while the contrast $C_\mathcal {B}$ of $\mathcal {B}$ is the average of the c(x)’s over the pixels of $\mathcal {B}$. The multi-resolution contrast C of $\mathcal {B}$ is the average of the contrasts of a set of images $\mathcal {B}_1 := \mathcal {B}, \ldots , \mathcal {B}_K$ ($K > 1$) where for each $1 = 2, \ldots , K$, $\mathcal {B}_i$ is the image $\mathcal {B}$ re-scaled by $2^{i-1}$. The higher C, the higher visibility of the details of $\mathcal {B}$ and of ${\overline{I}}$ is;

• 3. Histogram Flatness F: this is a measure of the entropy of the distribution of the image brightness $\mathcal {B}$; precisely, F is the $L^1$ distance between the probability density function (pdf) of $\mathcal {B}$ and the uniform pdf defined over the variability range of $\mathcal {B}$ (that here is assumed to be [0, 255]). The pdf of $\mathcal {B}$ is computed by normalizing the histogram of to sum up to 1.0. Low (high, resp.) values of F are typical of images with a high (low, resp.) dynamic range and thus with good (poor, resp.) detail visibility;

• 4. Image Regularity Measures $N_1$ and $N_2$: NIQE ($N_1$) [18] and BRISQUE ($N_2$) [17] are perceptual metrics assessing the level of naturalness of a color image ${\overline{I}}$ (basically, its local smoothness) by comparing some second-order statistics of ${\overline{I}}$ against those of a training set. While the training set of NIQE contains also regular images, that of BRISQUE includes pictures classified by human observers as regular or irregular. The lower these metrics are, the higher the image naturalness is.

Fig. 3.

Three images from SCA-30 (a) and their versions enhanced by GREAT-Mix (b), STRESS (c), STRESS-P (d) and GREAT (e).

It is to note that the exact amount of $B, C, F, N_1, N_2$ depends on the visual characteristics of the image ${\overline{I}}$. In particular, for already clear (poor readable, resp.) images, the variations of these measures after enhancement will be negligible (remarkable, resp.). In general, a contrast enhancer is expected to increase the value of C and F and to maintain unchanged or to decrease those of $N_1$ and $N_2$.

Table 1 reports the mean value of B, C, F, $N_1$, $N_2$ for the original images of SCA-30 (‘INPUT’) and for their versions enhanced by GREAT-Mix, STRESS, STRESS-P and GREAT. In this experiments, the parameter $\tau$ of GREAT and GREAT-Mix has been set as in Eq. (4), while the parameters n and p of STRESS and STRESS-P are set respectively to 25 and 100. The lightness values, that in the previous Sections have been assumed to range over (0, 1], are here re-scaled over [0, 255].

Table 1. Evaluation table

On average, GREAT, STRESS, STRESS-P and GREAT-Mix report similar values of C, F, $N_1$ and $N_2$, while GREAT produces brighter images. This is mainly due to the stretching operation, that maps to zero the intensities equal to the lower bound and thus decreases B. It is also easy to prove that $L_\mathrm{GREAT}(x) \ge L_\mathrm{i}(x)$ for any $x \in S$ and i = GREAT-Mix, STRESS. Both C and F are increased by enhancement, meaning that the algorithms considered here effectively improve the visibility of the image details while make the color distribution more uniform. The values of $N_1$ and $N_2$ after enhancement are on average slightly higher than those reported before enhancement: the distortions introduced by the enhancers are however exiguous and mainly due to an over-enhancement of noisy pixels located in low-light regions.

Table 1 also reports for each enhancer its average execution time per image (T). These data refer to a C++ implementation of the enhancers running on a standard PC with an Intel(R) Xeon(R) processor working at 3.70 GHz. Their comparison shows that the computational time of GREAT-Mix is very close to those of STRESS and GREAT, while it is remarkably smaller than that of STRESS-P: in this respect, GREAT-Mix bates STRESS-P since it provides similar, noise-free, repeatable results but with a much lower execution time. It is to note that no code optimization has been considered (e.g. Mi-Retinexes could be parallelized), thus the times reported in Table 1 may be further reduced. Nevertheless, such an optimization is out of the scope of this work.

Another advantage of GREAT-Mix versus STRESS and STRESS-P is the automatic estimation of its parameter $\tau$, which is inherited from GREAT and that avoids the trial-and-error procedure used to set the values of n and p.

Figure 3 shows some examples of enhancement. While for the images on left and right, GREAT-Mix provides results close to those of STRESS and STRESS-P, for the image in the middle, the output of GREAT-Mix is more similar to that of GREAT than to those of STRESS and STRESS-P. This is due to the ‘mix’ of computational blocks of GREAT and STRESS performed by GREAT-Mix.

Fig. 4.

Examples of contrast enhancement by GREAT-Mix applied to cultural heritage. (a) Three pictures showing from left to right: a particular of a paint in the cathedral of S. Vigilio in Trento, IT; a romanic arch and a mosaic in the Spazio Archeologico Sotterraneo del Sas (S.A.S.S.), Trento, IT; (b) Enhancement of the first two pictures in (a, left, middle) and of the particular of the mosaic highlighted in (a, right) by the red square. (Color figure online)

5 Conclusions

Originated from STRESS and GREAT, GREAT-Mix is a novel, noise-free, spatial color algorithm working as contrast enhancer. Future work will investigate possible applications of GREAT-Mix to cultural heritage. In this context, contrast enhancement is usually needed to study artworks that due to their location or structure cannot be captured by flash or cannot be well illuminated in all of their parts, making hard the work of archeologists and restorers (see Fig. 4). Contrast enhancement may contribute to cultural heritage studies and preservation by enabling the visual inspection and processing of digitalized artworks, that may help archeologists, restorers and scholars to plan their activities.