Aesthetic Interactive Hue Manipulation for Natural Scene Images

Abstract

One of the most common ways to adjust the aesthetic quality of a natural scene image is by editing its color. However, while lightness and saturation manipulation is well understood in computational terms, hue manipulation is poorly understood and non-trivial to affect. As a result, modifications made to hue with existing photo-editing tools often impart an unnatural look to the image. In this paper, we discuss a framework for hue manipulation and its contribution to the aesthetics of an image inspired by the use of hue in the visual arts. Our framework is designed to work with a segmentation approach based on superpixels that can segment the image into regions of similar color at varying scales. This allows the user the ability to quickly select semantically similar regions. These local regions also provide the basic region of operation for three proposed hue operations: (1) hue spread; (2) hue compression; and (3) hue shift. We show that when combined with superpixel segmentation these operations are capable of increasing, decreasing and changing regions of local hue contrast in a manner that produces natural or artistic looking images. We demonstrate our framework on a variety of input images and discuss its evaluation from the feedback of several expert users.

1 Introduction and Motivation

The ability to aesthetically improve a photograph has long been a popular subject. Software tools such as Photoshop¹ and GIMP² provide users a wide range of image editing functions to manipulate the appearance of an image. Most popular adjustments are those made to lightness and saturation values in order to modify the global or local contrast and vividness of the image respectively. The modification of hue, however, is more challenging and less understood.

Fig. 1.

Results of the three defined hue operations. Top row shows the original image, hue compressed image, hue stretched image and hue shifted image. Bottom row shows the corresponding hue histograms (Color figure online).

To better understand hue, we turn our attention to methods traditional painters use to organize the formal attributes of their paintings in such a way as to maintain a particular aesthetic quality. This is done to gain insight into how such organization can be applied in a computational manner. Similar approaches have been taken when examining lightness and saturation contrast. For example, Zhang et al. [16] compared the difference of lightness and saturation contrast within the artistic concept of the four depth regions. Using paintings as reference they developed a computational method to enhance the lightness and saturation contrast of photographs.

While lightness contrast and saturation contrast are fairly well understood as computational problems [6], hue is unique for the close relationship it has to the meaning of an object [10]. For example, foliage needs to be green, tomatoes needs to be red. Edits to hue risks damaging this relationship. Another problem is that there is little discussion or agreement as to what constitutes hue contrast.

Johhanes Itten was a Swiss artist and theorist whose work is well recognized in the art community. He identified a set of seven color contrasts [5] by which the colors of an artwork may be evaluated. Of these seven contrasts, three are hue specific. Two refer to the property of hue values on opposite sides of the hue wheel: the contrast of temperature and the contrast of complimentaries. The third, described as contrast of hue, refers simply to the property of there being many colors in an image or a region of an image.

Painters habitually variegate local regions of contiguous color so that they extend to include neighboring hue values. Hence the green of a tree would be painted so that it also encompasses yellow and blue, which border green on either side of the color spectrum. By either reducing or increasing this contrast the aesthetic properties of an image may be acted upon. Supporting this the art theorist Rudolph Arnheim wrote that: configuration of colors will strive either toward contrast or toward assimilation [2]. Figure 2 shows a detail of a painting by the painter William Finch (1854 – 1930) that shows a region of green grass being extended to cover the yellow and blue values. This effect may be understood as being an increase of Itten’s hue contrast. Conversely, a region of naturally variegated hue may be decreased and rendered as a single coherent hue. An example of such is shown in Fig. 2 in the painting ‘Western Motel’ by Edward Hopper.

Contribution. Inspired by Itten’s description of hue contrast, we propose an interactive image editing framework that allows the user to perform three operations for aesthetic hue manipulation. These are hue compression, stretching, and shifting (see Fig. 1), the first two being broadly analogues to an increase and decrease respectively of Itten’s contrast of hue. As discussed above, hue manipulation is more appropriately defined for local image regions and therefore needs to be coupled with image segmentation. To this end, we incorporate a superpixel-based interactive segmentation method that allows the user to easily select image regions at different superpixel scales (see Fig. 6). These segmented regions not only give users a way to select objects in the image content, but also provide the basic local region for hue manipulation as shown in Fig. 8.

The remainder of this paper is organized as follows: Sect. 2 describes work related to our framework. Section 3 describes our overall framework. Section 4 shows several results generated with our framework as well as comparisons with results from existing image-editing software. Section 5.2 provides feedback from several expert users in image-editing and color manipulation who have used our system. Section 6 concludes this paper with a short summary.

Fig. 2.

Edward Hopper, ‘Western Motel’, 1957 and Alfred William Finch, ‘Orchard at La Louvière’, 1890. Note the contiguous color of detail with subtle hue contrast in the first painting and variegated green with additional blue and yellows to give notable hue contrast in the second (Color figure online).

2 Related Work

2.1 Hue Manipulation

Although the manipulation of hue is well known and well adopted by artists, it has seen relatively little focus in image editing. Probably the best known hue manipulation is the creation of color harmony within an image. These approaches manipulate input hue such that values fit to a hue distribution that are considered to comprise of harmonic color sets. Well known examples include, Cohen-Or et al. [3]’s method that uses a template-driven approach to the harmonization of images derived from models developed by Matsuda [7] and Tokumaru et al. [14]. Wang et al. [15] proposed a knowledge-based system which adopts established color design rules into a comprehensive interactive framework to aid users in the selection of colors for scene objects. Nishiyama et al. [9] used a similar color harmony idea for aesthetic quality classification of photographs to obtain better aesthetic results. While these approaches produce impressive results through hue manipulation, they rely on established harmonic distributions and do not give as much flexibility to hue manipulation as our proposed method.

The most similar approach to our work is that by Constable et al. [4]. In this work, the authors mapped the hue histogram of their images onto a color wheel. If mapped in this manner the combined angle of spread reflected the total amount of perceived hue variety. This was used as an indicator of hue contrast: a large hue spread angle indicating high hue contrast and a small one indicating low. The advantage of defining hue contrast in this way is that it is made computationally available and that it can therefore be quantified and changed to the aesthetic benefit of the image.

2.2 Image Editing Software

There is a number of image editing software solutions that allow hue manipulation. For example, Adobe Photoshop and GIMP give the function to change the hue of an image through hue shifting. These are generally very hard to control and can easily impart an unnatural look to an image. More specific software solutions exist. HueShifter³ is a tool created specifically to change the hue of a region of an image in a manner that does not affect its perceptual lightness. Other kinds of software solutions, such as Color Schemer⁴ and Color Wheel Expert⁵, although not able to process an image directly, nonetheless provide a tool by which a harmonious color set could be designed that has been based on traditional color theory. Compared to such prior work, our framework is unique in its coupling of segmentation and the three hue manipulations: compression, stretching, and shifting using a single unified interface.

3 Hue Manipulation Framework

In this section we present the core concepts of our approach.

3.1 System Overview

Saturation and lightness values can be defined as pixel-wise variations of ‘amount’. Manipulation of these values act upon this amount. Hue can be measured as an amount if regarded as range of increasing wavelength along a short sequence of the electro magnetic spectrum. However, a more conventional and perceptually correct way of presenting hue is one where the two ends of this sequence are bent round to join each other forming the color wheel [8]. This arrangement reflects the non-hierarchical nature of hue, with no single hue term being greater than another. Therefore, hue can not be operated upon in the same manner as lightness and saturation. A novel approach is therefore needed.

Fig. 3.

Possible operations of our hue transfer system (Color figure online).

Our framework combines two main operations: image segmentation and hue adjustment. Figure 4 shows a screenshot of our software framework. As discussed earlier, our method provides three operations for hue: hue compression, hue stretch and hue shift. These three operations do not need to be used independently. As can be seen in Fig. 3, they can be combined to produce rich aesthetic results. As will be shown in Sect. 4, combining hue compression or stretching with hue shifting on different regions can significantly change the aesthetic appearance of an image. We chose to perform our color operations in the HSI color space. This color space presents hue as a separate channel in a manner that has its historic roots in the Munsell color system, wherein color is separated into its perceptually relevant components. HSI was derived from RGB color space in the following manner:

$$\begin{aligned} \alpha =\frac{1}{2}(2R-G-B), ~ \beta =\frac{\sqrt{3}}{2}(G-B),~ H=atan2(\beta ,\alpha ), \end{aligned}$$

$$\begin{aligned} m=min(R,G,B),~ I=\frac{1}{3}(R+G+B),~ S&=\left\{ \begin{array}{ll} 0 &{} \quad if~I = 0,\\ 1-\frac{m}{I} &{} \quad otherwise \end{array} \right. \!, \end{aligned}$$

(3.1)

where R, G, B are red, green and blue values in the RGB space and H, S, I are the hue, saturation and brightness values in the HSI space, respectively. Note that is possible to use other color spaces that present hue as a unique channel, such as HSV and HSL.

We note that the hue value becomes more ambiguous as the saturation and brightness decreases. For brightness and saturation values equal to zero, the value of hue may not be meaningful. As such, we only process the pixels with saturation (S) and brightness (I) larger than a certain threshold. Here we define the threshold to be $t_{s} = 0.10$ and $t_{i} = 0.15$ (assuming that S and I are within the range [0, 1]), respectively.

We discuss each component of our framework in detail in the following sections.

Fig. 4.

The user interface developed for the hue transfer.

3.2 Image Segmentation

As discussed in Sect. 1, hue manipulation is best defined in a local region of an image. This means that it will be necessary for the user to segment the image for hue processing. To this end, we use a segmentation strategy based on superpixels. In particular, we use the SLIC method [1] for superpixel segmentation that is spaced using color information. As a result, each superpixel semantically contains the pixels that are clustered together by their similarity of color and location. This is in agreement with human perception, since the items that we are considering are regions defined by likeness of hue and location, and not individual pixels. The user can select regions by simply drawing a scribble through the super pixels they want to select. Moreover, our system allows the user to control the size of the superpixels through a simple parameter selection. Larger superpixels increase the speed in which objects can be selected, whilst smaller superpixels provide access to finer grain regions. Examples of different superpixel segmentation are shown in Fig. 5.

The final superpixel segmentation will provide the basic unit of hue manipulation. That is, hue manipulation is applied to each superpixel independently.

Fig. 5.

Top: segmentations of size 30 and 100 (regularizor is 1). Bottom: segmentations with regulators 0.1 and 0.8 (size 100), respectively.

Fig. 6.

Top: shows the process of superpixel selection (selected super pixels are highlight in green on the right column). Bottom: shows the hue histogram of the whole image and the selected region (Color figure online).

Fig. 7.

Global hue transfer result. From left: input, hue compress, hue spread. In the bottom row are the corresponding hue histograms.

3.3 Hue Histogram Selection

As with existing image editing software such as Photoshop or GIMP, our tool allows users to manipulate a range of hue in a selected region. To isolate the hue range, the user can select a pixel in the image the hue value of which will be shown on the color wheel. A default interval centred at that hue value will be selected for the next step of hue transfer (we set $15^{\circ }$ distance from both sides of the indicated hue value by default), since our hue transfer operations are processed for a range of hues. The user can also manually adjust the two sides of the interval being processed by adjusting the position of the starting hue, denoted as A and ending hue, denoted as B on the hue wheel. A simple global hue transfer result (by the algorithms in Sect. 3.4) is shown in Fig. 7. Our system will automatically calculate the hue histogram of the selected region and denote it in the hue ramp using the markers A and B, which facilitates the next step of hue transfer. A local hue transfer result can be seen in Figs. 7 and 8 (by the algorithms in Sect. 3.4). Following the automatic selection of A and B, users can adjust markers to obtain accurately the part of the histogram they want to process. This can also be used to filter the part of the histogram they want to keep untouched. With the idea of superpixels, we can locally deal with hue in a meaningful way and process different parts of an image independently.

Fig. 8.

Sky and earth are processed independently due to local hue transfer. (superpixels are selected as in Fig. 6).

Fig. 9.

Hue shift can change the background of the image without affecting the foreground.

3.4 Hue Transfer Operators

After selecting a region of interest (or the whole image in the case of global transfer), users can decide to process one of the three designed operators: hue compress, hue stretch or hue shift. We provide two categories of operations: hue transfer by setting scale parameters and by adjusting interval (see Fig. 10).

Fig. 10.

Hue transfer by setting scale parameters and adjusting interval. The first, second and third columns are hue compression, stretching and shifting, respectively.

As indicated in previous steps in Sect. 3.3, markers A and B denote the two sides of the region that will be processed. Resulting interval is denoted by markers $A'$ and $B'$. Without losing generality, we consider that the hue axis is infinitely long with periodicity of $360^{\circ }$, and $A<B$, $A'<B'$ on the hue axis (e.g. Fig. 10). These 3 hue operators work by referring to the center of interval which is $\frac{A+B}{2}$.

In the case of hue transfer by setting scale parameters, for hue compress and hue stretch users can input a scale t that reflects the degree that the selected part of the hue histogram is compressed or stretched. The resulting $[A',B']$ is calculated as $A'=\frac{A+B}{2}-t\frac{B-A}{2}$ and $B'=\frac{A+B}{2}+t\frac{B-A}{2}$. Moreover, to make the two operations meaningful, t needs to be non-zero for hue compress and to be smaller that certain value so that $B'-A'$ will not be larger than $360^{\circ }$ for hue stretch. For hue shift users can input a shift amount C reflecting the color that they want to shift the selected region to. The resulting $[A',B']$ becomes $[A+C,B+C]$.

For hue transfer by adjusting interval [A, B], users give directly the positions of $A'$ and $B'$ on the hue wheel. A 1-D Example for the 3 operators can be seen in Fig. 10. Performing hue transfer by setting scale parameters gives users the roughly quantitative control on the hue histogram, whereas performing hue transfer by adjusting interval gives users the possibility to fine tune the hue.

Fig. 11.

Left: an image of a flower bed before treatment. Middle: after hue compression, note the lessening of the variety of greens. Right: after hue stretch. Note the increase of the variety of greens that spreads into yellow (Color figure online).

Given the hue values H(x, y) of pixel (x, y) defined in the region of interest whose histogram is bounded by A and B, hue transfer aims to transform the current histogram of the hue function H(x, y) within [A, B] to a new histogram of interval $[A',B']$. We update the hue values $H'(x,y)$ using the following rule:

$$\begin{aligned} H'=\frac{B'-A'}{B-A}(H-A)+A'. \end{aligned}$$

(3.2)

The modulo operation is needed to make sure that the final hue value $H'$ is located within the meaningful range [0, 360].

4 Results

The following shows several results from our software. In Sect. 5.2 we provide feedback from expert users on their experience with hue operations and their opinions of our software.

4.1 Hue Shift, Compress and Stretch

Hue shift can be used to change the emotional ambience of a photograph. For example, a warm color is often used in the foreground against a cool background to give a harmonious result. Alternately, the local contrast between two or more different semantic parts of a photograph may be increased to improve its quality. An example of a hue shift result can be seen in Fig. 9.

Hue compression has the aesthetic effect of lessening the differences in hue within local regions (middle image of Fig. 11). This was very different from what could be achieved with the tool in stretch mode (bottom image of Fig. 11). Using a hue compression, the naturally variegated appearance of a tropical jungle can be compressed so that it more closely resembles the more monolithic hue of a European forest (Fig. 1).

Fig. 12.

Examples of compound hue transfer operations.

The ideal image for the hue stretching and compression is one where there is an existing hue variety within the target region. This is in line with expectations, hue stretching and compression being conceptually impossible within regions of hue homogeneity. Hue stretch and compress produced some noticeable results when it was applied to such images (in Figs. 1, 7 and 11). Scenes featuring foliage, rocks, soil and other natural phenomenon are excellent examples of scenes with strong hue variation that can be amplified through hue stretching. Most man-made materials, being generally homogeneous in their hue values, do not make good subjects for our approach. However, those that displayed a degree of hue variation produced some interesting though unnatural results.

Fig. 13.

Examples of hue transfer compared to Photoshop. Row 1 shows the results after hue stretch. Row 2 shows the results after hue compression. Row 3 shows the result after hue shift.

Generally good hue stretch results were possible if the target hue range did not span more than one hue term and if the degree of stretch did not also span more than one hue term on either side of the hue spectrum for one object or one group of similar objects. This is again in line with expectations regarding the importance of hue terms to the semantic reading of an image.

4.2 Combining the Three Operations

In Fig. 3 we can see that by a combination of hue stretch and hue shift a variety of color effects can be derived which together makes the scene look like summer or autumn.

Our image processing method by hue can be decomposed to a series of hue transfer operations. To each operation can be attached semantic significance since color has a close relation with human experience. In Fig. 12 we show other examples of combining the three operations.

5 Discussion

5.1 Comparison with Other Software

Some of the existing image editing software, though to some extent allow hue to be adjusted separately to saturation and lightness, but none of them offer the regional and contextual control over hue spread that our approach offers.

Almost all of the state-of-the art of color manipulation methods are example-based (e.g. [11–13, 16]), which means a reference image is needed. Different from them, our method requires user interaction. Here we compare our method with the existing commercial bitmap editor Photoshop CS6. Other software, such as GIMP can perform similar tasks with slightly different operations. Here we specifically compare their hue processing functions.

To test our result we invited some professional Photoshop artists to attempt to match our results using Photoshop. For the example of hue stretch, they were able to do reasonable job, however their results did not match ours in the subtlety and richness of the hue variegation across the tonal range. Importantly, their results required a complex multilayered approach. For one good achieved result shown in Fig. 14, in all, the artist used 4 layers and 2 masks and the whole operation required 9 separate steps. To control the effect spilling over into the sky region, it was necessary to mask out the effect. Note that our result is produced by the global hue transfer, without selecting out some specific region by superpixels to further carry out processing.

Three more comparison examples regard to hue compress, stretch and shift are shown in Fig. 13 in such order. The results compared to were done by some artists using the hue functions provided by Photoshop. Here the 3 hue transfers were done globally. We can see that our results are more natural and give more harmonic effects than those achieved with Photoshop when both of the two method only processes the hue.

Some commercial software, such as Lightroom⁶, provide individual sections of the hue range manipulation where user can shift hue within some defined operations. However, such kind of hue operations are still limited and different from our idea of manipulating the hue histogram and hue contrast.

Fig. 14.

Comparison of our result and one of Photoshop with complicate operation. The Photoshop result required 4 layers and 9 separate operations to achieve. Ours were done globally and required only one operation after well selecting interested region [A, B] and transferred region $[A',B']$.

5.2 User Feedback

Additional feedback was gathered from a group of ten professionals in the filmmaking postproduction industry. These professionals were used to using a wide range of professional software, much of it developed in-house. Amongst this group was one who supervised an Oscar award winning visual effects team and another whose work was recognised with a prestigious Annie award. They therefore constitute a very capable group of users who operate at the very highest level of professional practice. The principles behind the tool were described to them and a sample of results was shown. They were asked if they thought the tool would be useful and if they could reproduce the same effects using existing tools. All of them were convinced of the usefulness of our tool and that its function was not naturally served by any existing solutions. Results achieved using such methods usually require masking before they are usable. Additionally, hue expansion is more achievable than hue compression. Others thought that in the right circumstances a hue replace would suffice if moderated with a mask. If a slight hue spread or compression was needed, then it could sometimes produce good results. Neither of the suggested approaches were as intuitive to manage as ours and both of them would be difficult to effect in Photoshop (RGB to HSI conversation is not naturally supported, though optional plugins are available).

6 Conclusion

We have presented an image editing framework to adjust the aesthetic quality of an image by manipulating its hue. Modifications made to hue with existing photo-editing tools can easily impart an unnatural look to the image. Considering that hue is poorly understood and non-trivial to affect, we propose three hue transfer operations for color manipulation. To realize our idea, we developed a user interaction tool that incorporates image segmentation to provide the user the possibility to deal with hue in a flexible and powerful way. The before and after values of the hue adjustment are presented in an intuitive and easy-to-understand manner. We demonstrate the effectiveness of our methods by comparing the results with those produced using Photoshop.