Encyclopedia of Color Science and Technology

2016 Edition
| Editors: Ming Ronnier Luo

Retinex Theory

  • John McCann
Reference work entry
DOI: https://doi.org/10.1007/978-1-4419-8071-7_260

Synonyms

Definition

Retinex is the theory of human color vision proposed by Edwin Land to account for color sensations in real scenes. Color constancy experiments showed that color does not correlate with receptor responses. In real scenes, the content of the entire image controls appearances. A triplet of L, M, S cone responses can appear any color. Land coined the word “Retinex” (the contraction of retina and cortex) to identify the spatial image processing responsible for color constancy. Further, he showed that color sensations are predicted by three lightnesses observed in long-, middle-, and short-wave illumination. Retinex is also used as the name of computer algorithms that mimic vision’s spatial interactions to calculate the lightnesses observed in complex scenes.

Overview

Edwin H. Land, the inventor of hundreds of film patents, was struck by experiments showing that color sensations in real complex images depend on scene content. Film responds to the light falling on each tiny local region. Land realized that vision’s mechanisms were very different from film. His early experiments studied the colors observed in red and white projections [1]. He realized color appearance required both the cone responses to a local region and the neural spatial processing of the rest of the scene. He proposed the Retinex Theory.

Land coined the word Retinex to describe three independent spatial channels. In 1964 he wrote: “We would propose that all of the receptors with maximum sensitivity to the long-waves in the spectrum, for example, operate as a unit to form a complete record of long-wave stimuli from objects being observed. (For convenience of reference, let us call this suggested retinal-cerebral system a “retinex.”)” [2, 3, 4, 5]. It is the word that describes the mechanism that performs the comparison of scene information to create the array of sensations of lightness in three channels.

Cone Quanta Catch

Visible light falls on objects that reflect some of it to the eye. Color vision depends on the spectrum of the illumination falling on an object and the spectrum of its reflectance. The product of these spectra describes the light coming to the eye. There are three types of cones in normal observers that are called L for long-wave-, M for middle-wave-, and S for short-wave-sensitive cones. The receptors’ spectral sensitivities multiplied by the light falling on the retina determines the L, M, S cone responses, namely, the “quanta catch” of the cones. The cones convert the quanta catch to nerve signals that pass through many spatial comparisons in the visual system. The cone quanta catch is the important transition from the physics of light to the physiology of vision. However, it is just the first step in the process.

Figure 1 illustrates a laboratory experiment that generates equal L, M, S quanta catches from different reflectance papers.
Retinex Theory, Fig. 1

Equal quanta catches from different papers

Light passes through filters that determine the spectrum of each illumination falling on circular pieces of paper. The light coming to the eye from the papers is modified again by the reflectance spectra of the papers. Further, in this experiment there is no light coming from the black surrounds.

On the left, there is a tungsten light source at the top. There is a filter that absorbs more middle-wave than long- and short-wave visible light (magenta arrows). The green circular paper reflects more middle-wave than long- and short-wave light. The light coming to the eye is the product of these spectra. The integrals of that light using the three cone spectral sensitivities determine the L, M, S quanta catch values.

On the right, there is the same tungsten light source at the top. There is a different filter that absorbs more long-wave than middle- and short-wave light (cyan arrows). The right-red paper reflects more long-wave than middle- and short-wave light. In this experiment, the spectra of the two illuminants and the two reflectances were adjusted to generate the same triplet of LMS cone quanta catches. Under these conditions, the left-green and right-red papers are identical retinal stimuli. They appear equal to each other, but that color is neither red nor green.

Mondrian Experiments

Land’s color Mondrian experiment is similar, with the exception that he used a complex array of papers to simulate real-world scenes. The important difference is that in complex scenes, a particular quanta catch can appear in any color: red, green, blue, yellow, white, or black.

Figure 2 shows the double color Mondrian experiment [6]. It used two identical Mondrians made of color papers, and three different, non-overlapping spectral illuminants (long-, middle-, and short-wave visible light). In this experiment observers reported the colors of papers in the Mondrians. In this illustration, we will look at the circular red and green papers. Land adjusted the illuminant mixtures of light from the two sets of three projectors. The same amounts of L, M, S light came from the green circle on the left as from the red circle on the right. First, he turned on just the long-wave lights. He adjusted the amounts of illumination on the left-green and right-red circles so the meter readings were equal. Then, he did the same for middle- and short-wave light. The left-green and right-red circles had equal quanta catches by the L, M, S cones.
Retinex Theory, Fig. 2

Land’s double Mondrian experiment. Two identical sets of matte colored papers, with separate L, M, S illuminating projectors with voltage transformers for control of the amount of light. Telephotometer readings were projected above the Mondrians. The experimenter separately measured the L, M, S radiances from a green circle in the left Mondrian. Then, he adjusted the L, M, S radiances from a red circle on the right Mondrian to be the same. Observers reported different red and green colors produced by identical light stimuli

In this complex scene, observers reported that equal quanta catches appeared green on the left and red on the right. Observers reported color constancy, namely, that the red paper looked red and the green paper looked green despite the identical cone quanta catch.

Land repeated this experiment with all the Mondrian papers. A constant L, M, S quanta catch could generate any color sensation. The presence of the complex scene introduced more information to the visual system. The red and green papers appeared equal in Fig. 1. The red paper looked red and the green paper looked green in Fig. 2. The scene’s spatial content stimulated vision’s spatial image processing mechanisms to generate color constancy. The post-receptor visual processing plays a dominant role in color appearance in real scenes. Land’s word “Retinex” gave this spatial process a name. As well, he proposed a theoretical mechanism.

Retinex Mechanism

Figure 3 illustrates the pair of Mondrians in only long-wave light with more light on the left than on the right. Their appearances are nearly constant. This is a common observation: humans are insensitive to large changes in uniform illumination.
Retinex Theory, Fig. 3

Pair of Mondrians in only long-wave light. The left Mondrian has more illumination than the right. Observers report that the left set is slightly lighter than the right. Each corresponding area is nearly the same lightness in the left and right Mondrians

As illustrated here, the left-green circle looked dark when it generated the same L cone quanta catch as the lighter right-red circle. Vision’s spatial image processing rendered the red and green papers with different lightnesses in long-wave light. The lightnesses are stable with large changes in overall illumination.

Figure 4 illustrates the Mondrian viewed in only middle-wave light. The left-green circle now looks light and the right-red looks dark. In the experiment they had the same M cone quanta catch. Vision’s spatial image processing rendered the red and green papers with different and opposite lightnesses in Fig. 4.
Retinex Theory, Fig. 4

Pair of Mondrians in only middle-wave light. Now, the left Mondrian has less illumination than the right. In this wave band the pattern of lightnesses differs from that in long-wave light. That lightness pattern is indifferent to the amount of uniform illumination. Land adjusted the left side and right side illumination so that the circles had equal meter readings

In summary, if the two Mondrians are side by side in the same band of wavelengths, but different overall intensities, the observers report nearly the same set of lightnesses at corresponding locations in the left and right Mondrians. However, one side is detectably lighter than the other. With large uniform changes in illumination, observers report nearly constant lightnesses of the individual papers.

The left-green paper has more long-wave and less middle-wave illumination. The right-red paper has less long-wave and more middle-wave illumination. When adjusted, those adjustments in amount of illumination make the red and green papers have identical radiances. Those adjustments do not significantly alter the lightnesses of the areas in separate illumination. When viewing the Mondrian in combined illumination, in color, those changes in illumination do not change the color appearances of the red and green papers.

These observations led Land to propose the Retinex theory. The triplet of apparent lightnesses, not cone quanta catches, determines the color appearance. Constant LMS lightnesses generate constant colors. That hypothesis led to a study of color appearances in L, M, S bands of light. Do all red colors have the same triplet of lightness appearances? Does a red color always look [light, dark, dark] in L, M, S light? Does a green always look [dark, light, dark]? Does color appearance always correlate with the triplet of L, M, S lightnesses?

The experiment is easy. Find a red, a green, and a blue filter. Be sure that the filters exclude the other two-thirds of the spectra. With a green filter you should just see greens with different lightnesses. You should not see a mixture of greens and yellows and blues. If you do, you need a filter with a narrower band of transmission.

Identify a group of red objects. Look at them sequentially through the L, M, S filters. Look at them in different ambient illuminations. Look at them at different times of the day. Look at them in sunlight and shadows. Red colors are always (light, dark, dark) in L, M, S light. The same dependence of the triplet of L, M, S lightnesses holds for all colors (Table 1). Lightness is the output of spatial image processing. It is the result of post-receptor spatial processing. That is why lightness does not correlate with cone quanta catch. However, color does correlate with three lightnesses in long-, middle-, and short-wave light.
Retinex Theory, Table 1

Correlation table of color appearances and the apparent lightnesses in L, M, S illumination

Retinex theory predicts that the triplet of L, M, S lightnesses determines color. Colors are constant with changes in illumination because the triplet of lightnesses is nearly constant.

Land’s observation still stands: The triplet of lightnesses correlates with color. The observation is important because a variety of different phenomena can influence lightness, such as simultaneous contrast, the Cornsweet effect, and assimilation. Regardless of the cause of the lightness changes, when two identical physical objects look different, color appearances correlate with their L, M, S lightnesses ([7], Section E).

In the color assimilation display (Fig. 5), there are two sets of nine red squares that have the same reflectance and appear the same (top left). However, if these red squares are surrounded by yellow and blue stripes, they look different (top center): the left red squares fall on top of the yellow stripes, and the right ones on the blue stripes. The left squares appear a purple red, while the right ones appear a yellow orange. In other words, the left squares appear more blue and the right ones more yellow.
Retinex Theory, Fig. 5

(Top-left) Color squares and (top-right) color assimilation change their appearance; (bottom) L, M, S separation images. Color appearances of red squares correlate with L, M, S lightnesses

In the L separation the corresponding squares are lighter on the right of the separation; in the M separation these patches are lighter; in the S separation they are darker on the right. Land’s Retinex predicts that whenever L and M separations are lighter and S separation is darker, then that patch will appear more yellow. Whenever S separation is lighter and L and M separations are darker, then that patch will appear more blue. Colors correlate with L, M, S lightnesses ([7], pp. 221–281).

Retinex Image Processing

Land described that the fundamental challenge of color vision shifted to the ability to predict lightness; that is, the spatial interactions found in post-receptor neural processes. In 1967 Land and McCann proposed a computational model for calculating lightness from the array of all scene radiances [6]. The model compared each pixel with every other pixel in an image. The goal was to calculate the sensation of image segments that equaled what observers saw. In the past 50 years, there have been many implementations and variations of this process. They are called Retinex algorithms. It is curious that Land reserved the use of the term “Retinex” to describe three independent lightness channels. Today’s usage of the word includes a much wider range of computer algorithms that build calculated appearances out of arrays of radiances.

To calculate lightnesses in complex scenes, one must:
  • Capture scene radiances.

  • Convert scene radiances to cone and rod quanta catches.

  • Calculate lightness using all pixels in the scene.

  • Compare calculated lightness with observer matches.

The Land and McCann model [6, 9, 11] used:
  • Edge ratios

  • Gradient threshold (found to be unnecessary in later studies)

  • Multiplication of edge ratios (made long-distance interactions)

  • Reset to maxima (scaled the output)

    (introduced dependence on scene content, e.g., simultaneous contrast)

  • Average of many spatial comparisons

The first computer implementation of the model used an array of 20 by 24 pixels. McCann, McKee, and Taylor showed that long-, middle-, and short-wave computed lightnesses predicted observer matches of color Mondrians in color constancy experiments [8].

Since the late 1960s, computer imaging has shown remarkable advances. Digital images have replaced film in most of photography. Computer graphics has made image synthesis ubiquitous. Retinex image processing has grown with the advances in digital imaging [9, 10, 11]. In the early 1980s Frankle and McCann introduced a multi-resolution algorithm that allowed efficient comparison of all pixels in the image [12]. Jobson and Kotera with their colleagues have studied the NASA Retinex. Rizzi and colleagues have developed the Milan Retinex ([7], pp. 324–328). Sobol extended that Retinex algorithm was used in the design of commercial cameras [13]. Other algorithms have used Retinex spatial processing in color gamut-mapping applications [14].

The important feature of real complex scenes is that the illumination is rarely uniform. Shadows and multiple reflections increase the dynamic range of light coming to our eyes and to cameras. The application of Retinex algorithms to high dynamic range (HDR) scenes has become a major topic of research and engineering applications. The limits of HDR scene capture and reproduction are controlled by optics, namely, optical veiling glare. Camera glare limits the range of light on the sensor, just as intraocular glare limits the range of light on the retina. The scene content controls the range of light in images. Vision’s post-receptor neural processes compensate for veiling glare. That explains humans’ high dynamic range of appearances from low-dynamic-range retinal images. The spatial mechanisms modeled by Retinex algorithms play a major role in compensating for glare and generating our range of color and lightness sensations.

Over the years many variations of spatial processing mimicking human vision have been called Retinex algorithms.

Cross-References

References

  1. 1.
    McCann, J.J., Benton, J., McKee, S.: Red/white projections and rod/long-wave cone color: an annotated bibliography. J. Electron. Imaging 13, 8–14 (2004)ADSCrossRefGoogle Scholar
  2. 2.
    Land, E.H.: The retinex. Am. Sci. 52, 247–264 (1964)Google Scholar
  3. 3.
    Land, E.H.: The retinex theory of colour vision. Proc. R. Inst Gr. Brit. 47, 23–58 (1974)Google Scholar
  4. 4.
    Land, E.H.: Smitty Stevens’ test of retinex theory. In: Moscowitz, H., et al. (eds.) Sensation and Measurement, Papers in Honor of S. S. Stevens, pp. 363–368. Applied Science Publishers, London (1974)CrossRefGoogle Scholar
  5. 5.
    Land, E.H.: The retinex theory of color vision. Sci. Am. 237, 108–128 (1977)CrossRefGoogle Scholar
  6. 6.
    Land, E.H., McCann, J.J.: Lightness and retinex theory. J. Opt. Soc. Am. 61, 1–11 (1971)ADSCrossRefGoogle Scholar
  7. 7.
    McCann, J., Rizzi, A.: The Art and Science of HDR Imaging, pp. 221–375. Wiley, Chichester (2012)Google Scholar
  8. 8.
    McCann, J.J., McKee, S., Taylor, T.: Quantitative studies in retinex theory, a comparison between theoretical predictions and observer responses to color Mondrian experiments. Vision Res. 16, 445–458 (1976)CrossRefGoogle Scholar
  9. 9.
    McCann J.J.: Lessons learned from Mondrians applied to real images and color gamuts. In: Proceedings of IS&T/SID Color Imaging Conference, vol. 7, pp. 1–8. Scottsdale (1999)Google Scholar
  10. 10.
    McCann, J.J.: Simultaneous Contrast and Color Constancy: Signatures of Human Image Processing, Chapter 6 in Color Perception: Philosophical, Psychological, Artistic, and Computational Perspectives, Vancouver Studies in Cognitive Studies: Davis, A., Davis, S. (eds.) pp. 87–101. Oxford University Press, Vancouver 2000Google Scholar
  11. 11.
    McCann, J.J.: Capturing a black cat in shade: past and present of retinex color appearance models. J. Electron. Imaging 13, 36–47 (2004)ADSCrossRefGoogle Scholar
  12. 12.
    Frankle, J., McCann, J.: Method and apparatus of lightness imaging. US Patent 4,384,336 (1983)Google Scholar
  13. 13.
    Sobol, R.: Improving the retinex algorithm for rendering wide dynamic range photographs. J. Electron. Imaging 13, 65–74 (2004)ADSCrossRefGoogle Scholar
  14. 14.
    McCann, J.J.: A spatial color-gamut calculation to optimize color appearance. In: MacDonald, L., Luo, R. (eds.) Colour Image Science: Exploiting Digital Media, pp. 213–233. Wiley, Chichester (2002)Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.McCann ImagingArlingtonUSA