Measuring constancy

This section illustrates how constancy may be measured by describing experiments conducted by Kraft and Brainard (1999; see also Brainard, 1998; Kraft et al., 2002). Before treating the specific experimental design, however, some general remarks are in order.

Several distinct physical processes can cause the illumination impinging on a surface to vary. The images in Figure 61.1 illustrate one such process. They were taken at different times, and the spectra of the illuminant sources changed. Color constancy across illumination changes that occur over time is called successive color constancy.

Geometric factors can also cause the illumination impinging on a surface to change. This is illustrated by Figure 61.2. All of the effects shown occur without any change in the spectra of the light sources but instead are induced by the geometry of the light sources and objects. Color constancy across illumination changes that occur within a single scene is called simultaneous color constancy.

Figure 61.2..  

Image formation. Each set of square patches around the side of the image illustrates variation in the light reflected to the eye when surface reflectance is held fixed. Gradient: The two patches shown were extracted from the upper left (L) and lower right (R; above table) of the back wall of the scene. Shadow: The two patches were extracted from the tabletop in direct illumination (D) and shadow (S). Shape: The three patches shown were extracted from two regions of the sphere (T and B; center top and right bottom, respectively) and from the colored panel directly above the sphere (P; the panel is the leftmost of the four in the bottom row). Both the sphere and the panel have the same simulated surface reflectance function. Pose and indirect illum: The four patches were extracted from the three visible sides of the cube (R, L, and T; right, left, and top visible sides, respectively) and from the left side of the folded paper located between the cube and the sphere (I). The simulated surface reflectances of all sides of the cube and of the left side of the folded paper are identical. The image was rendered from a synthetic scene description using the RADIANCE computer graphics package (Larson and Shakespeare, 1998). There were two sources of illumination in the simulated scene: a diffuse illumination that would appear bluish if viewed in isolation and a directional illumination (from the upper left) that would appear yellowish if viewed in isolation. All of the effects illustrated by this rendering are easily observed in natural scenes. (See color plate 37.)

The visual system's ability to achieve simultaneous constancy need not be easily related to its ability to achieve successive constancy. Indeed, fundamental to simultaneous constancy is some sort of segmentation of the image into regions of common illumination, while such segmentation is not obviously necessary for successive constancy (Adelson, 1999). Often results and experiments about successive and simultaneous constancy are compared and contrasted without explicit acknowledgment that the two may be quite different; keeping the distinction in mind as one considers constancy can reduce confusion. This chapter will focus on successive constancy, as many of the key conceptual issues can be introduced without the extra richness of simultaneous constancy. The discussion returns briefly to simultaneous constancy.

At the beginning of the chapter, constancy was cast in terms of the stability of object color appearance, and this is the sense in which the experiments presented below assess it. Some authors (Brainard and Wandell, 1988; D'Zmura and Mangalick, 1994; Foster and Nascimento, 1994; Khang and Zaidi, 2002) have suggested that constancy might be studied through performance (e.g., object identification) rather than through appearance per se. One might expect appearance to play an important role in identification, but reasoning might also be involved. Although the study of constancy using performance-based methods is an interesting direction, this chapter is restricted to measurements and theories of appearance.

An Example Experiment

Figure 61.3 shows the basic experimental setup used by Kraft and Brainard (1999). Subjects viewed a collection of objects contained in an experimental chamber. The chamber illumination was provided by theater lamps. The light from the lamps passed through a diffuser before entering the chamber, so that the overall effect was of a single diffuse illuminant. Each lamp had either a red, green, or blue filter, and by varying the intensities of the individual lamps, the spectrum of the chamber illumination could be varied. Because the light in the chamber was diffuse, the viewing environment provided a rough approximation to Mondrian World conditions.

Figure 61.3..  

Schematic diagram of the experimental apparatus used in the experiments of Kraft and Brainard. An experimental chamber was illuminated by computer-controlled theater lamps. Different filters were placed over individual lamps, so that by varying their relative intensity the overall spectral power distribution of the chamber illumination could be varied. The light from the lamps was passed through a diffuser, producing a fairly homogeneous illumination. The observer viewed a test patch on the far wall of the chamber. The test patch was illuminated by the ambient chamber illumination and also by a beam from a projection colorimeter. The beam from the colorimeter was not explicitly visible, so that the perceptual effect of varying it was to change the apparent surface color of the test patch.

The far wall of the experimental chamber contained a test patch. Physically, this was a surface of low neutral reflectance so that under typical viewing conditions it would have appeared dark gray. The test patch was illuminated by the ambient chamber illumination and also by a separate projector. The projector beam was precisely aligned with the edges of the test patch and consisted of a mixture of red, green, and blue primaries. By varying the amount of each primary in the mixture, the light reflected to the observer from the test patch could be varied independently of the rest of the image. The effect of changing the projected light was to change the color appearance of the test, as if it had been repainted. The apparatus thus functioned to control the effect described by Gelb (1950; see also Katz, 1935; Koffka, 1935), wherein using a hidden light source to illuminate a paper dramatically changes its color appearance.

The observer's task in Kraft and Brainard's (1999) experiments was to adjust the test patch until it appeared achromatic. Achromatic judgments have been used extensively in the study of color appearance (e.g., Chichilnisky and Wandell, 1996; Helson and Michels, 1948; Werner and Walraven, 1982). During the adjustment, the observer controlled the chromaticity of the test patch while its luminance was held constant. In essence, the observer chose the test patch chromaticity, which appeared gray when seen in the context set by the rest of the experimental chamber. Whether the test patch appeared light gray or dark gray depended on its luminance. This was held constant during individual adjustments but varied between adjustments. For conditions where the luminance of the test patch is low relative to its surroundings, Brainard (1998) found no dependence of the chromaticity of the achromatic adjustment on test patch luminance. This independence does not hold when more luminous test patches are used (Chichilnisky and Wandell, 1996; Werner and Walraven, 1982).

The data from the experiment are conveniently represented using the standard 1931 CIE chromaticity diagram. Technical explanations of this diagram and its basis in visual performance are widely available (e.g., Brainard, 1995; CIE, 1986; Kaiser and Boynton, 1996), but its key aspects are easily summarized. Human vision is trichromatic, so that a light C(λ) may be matched by a mixture of three fixed primaries:

In this equation P₁(λ), P₂(λ), and P₃(λ) are the spectra of the three primary lights being mixed, and the scalars X, Y, and Z specify the amount of each primary in the mixture. The symbol ∼ indicates visual equivalence. When we are concerned with human vision, standardizing a choice of primary spectra allows us to specify a spectrum compactly by its tristimulus coordinates X, Y, and Z. The CIE chromaticity diagram is based on a set of known primaries together with a standard of performance that allows computation of the tristimulus coordinates of any light from its spectrum. The chromaticity diagram, however, represents lights with only two coordinates, x and y. These chromaticity coordinates are simply normalized versions of the tristimulus coordinates:

The normalization removes from the representation all information about the overall intensity of the spectrum while preserving the information about the relative spectrum that is relevant for human vision.

Figure 61.4 shows data from two experimental conditions. Each condition is defined by the scene within which the test patch was adjusted. The two scenes, labeled Scene 1 and Scene 2, are shown at the top of the figure. The scenes were sparse but had visible three-dimensional structure. The surface lining the chamber was the same in the two scenes, but the spectrum of the illuminant differed. The data plotted for each condition are the chromaticity of the illuminant (open circles) and the chromaticity of the observers' achromatic adjustments (closed circles).

Figure 61.4..  

Basic data from an achromatic adjustment experiment. The images at the top of the figure show the observer's view of two scenes, labeled 1 and 2. The test patch is visible in each image. The projection colorimeter was turned off at the time the images were acquired, so the images do not show the results of observers' achromatic adjustments. The chromaticity diagram shows the data from achromatic adjustments of the test patch made in the context of the two scenes. The open circles show the chromaticity of the illuminant for each scene. The illuminant for Scene 1 plots to the lower left of the illuminant for Scene 2. The closed circles show the chromaticity of the mean achromatic adjustments of four observers. Where visible, the error bars indicate ±1 standard error. The surface reflectance function plotted in the inset at the right of the figure shows the equivalent surface reflectance S ˜ (λ) computed from the data obtained in Scene 1. The closed diamond shows the color constant prediction for the achromatic adjustment in Scene 2, given the data obtained for Scene 1. See the explanation in the text. (See color plate 38.)

The points plotted for the illuminant are the chromaticity of the illuminant, as measured at the test patch location when the projection colorimeter was turned off. These represent the chromaticity of the ambient illumination in the chamber, which was approximately uniform. The fact that the illuminant was changed across the two scenes is revealed in the figure by the shift between the open circles.

The plotted achromatic points are the chromaticity of the light reflected to the observer when the test appeared achromatic. This light was physically constructed as the superposition of reflected ambient light and reflected light from the projection colorimeter. Across the two scenes, the chromaticity of the achromatic point shifts in a manner roughly commensurate with the shift in illuminant chromaticity.

Relation of the Data to Constancy

What do the data plotted in Figure 61.4 say about color constancy across the change from Scene 1 to Scene 2? A natural but misleading intuition is that the large shift in the achromatic locus shown in the figure reveals a large failure of constancy. This would be true if the data plotted represented directly the physical properties of the surface that appears achromatic. As noted above, however, the data plotted describe the spectrum of the light reaching the observer. To relate the data to constancy, it is necessary to combine information from the measured achromatic points and the illuminant chromaticities.

Suppose that the observer perceives the test patch as a surface illuminated with the same ambient illumination as the rest of the chamber. Introspection and some experimental evidence support this assumption (Brainard et al., 1997). The data from Scene 1 can then be used to infer the spectral reflectance of an equivalent surface. The equivalent surface would have appeared achromatic had it been placed at the test patch location with the projection colorimeter turned off.

Let the reflectance function of the equivalent surface be S ˜ (λ). This function must be such that the chromaticity of E₁(λ) S ˜ (λ) is the same as the chromaticity of the measured achromatic point, where E₁(λ) is the known spectrum of the ambient illuminant in Scene 1. It is straightforward to find functions S ˜ (λ) that satisfy this constraint. The inset to Figure 61.4 shows one such function. The function S ˜ (λ) is referred to as the equivalent surface reflectance corresponding to the measured achromatic point.

The equivalent surface reflectance S ˜ (λ) allows us to predict the performance of a color constant observer for other scenes. To a constant observer, any given surface should appear the same when embedded in any scene. More specifically, a surface that appears achromatic in one scene should remain so in others. Given the data for Scene 1, the chromaticity of the achromatic point for a test patch in Scene 2 should be the chromaticity of E₂(λ) S ˜ (λ), where E₂(λ) is the spectrum of the illuminant in Scene 2. This prediction is shown in Figure 61.4 by the closed diamond.

Although the measured achromatic point for Scene 2 does not agree precisely with the constancy prediction, the deviation is small compared to the deviation that would be measured for an observer who had no constancy whatsoever. For such an observer, the achromatic point would be invariant across changes of scene. Thus, the data shown in Figure 61.4 indicate that observers are approximately color constant across the two scenes studied in the experiment.

Brainard (1998) developed a constancy index that quantifies the degree of constancy revealed by data of the sort presented in Figure 61.4. The index takes on a value of 0 for no adjustment and 1 for perfect constancy, with intermediate values for intermediate performance. For the data shown in Figure 61.4, the constancy index is 0.83. This high value seems consistent with our everyday experience that the colors of objects remain stable over changes of illuminant but that the stability is not perfect.

A Paradox and Its Resolution

The introductory section stated that illuminant and surface information is perfectly confounded in the retinal image. The data shown in Figure 61.4 indicate that human vision can separate these confounded physical factors and achieve approximate color constancy. This presents a paradox. If the information is perfectly confounded, constancy is impossible. If constancy is impossible, how can the visual system be achieving it?

The resolution to this paradox is found by considering restrictions on the set of scenes over which constancy holds. Figure 61.5 shows a schematic diagram of the set of all scenes, represented by the thick outer boundary. Each point within this boundary represents a possible scene, that is, a particular choice of illuminant and surface reflectances. The closed circles represent the two scenes used in the experiment described above. These are labeled Scene 1 and Scene 2 in the figure.

Figure 61.5..  

Schematic illustration of the ambiguity inherent in color constancy. The figure shows schematically the set of all scenes. Each point in the schematic represents a possible scene. Scenes 1 and 2 from the experiment described in text are indicated by closed circles. Each shaded ellipse encloses a subset of scenes that all produce the same image. The scenes represented by open circles, Scenes 1 ˜ and 2 ˜ , produce the same images as Scenes 1 and 2, respectively. The open ellipses each enclose a subset of scenes that share the same illuminant.

Denote the retinal image produced from Scene 1 as Image 1. Many other scenes could have produced this same image. This subset of scenes is indicated in the figure by the shaded ellipse that encloses Scene 1. This ellipse is labeled Image 1 in the figure. It also contains Scene 1 ˜ , indicated by an open circle in the figure. Scenes 1 and 1 ˜ produce the same image and cannot be distinguished by the visual system.

Similarly, there is a separate subset of scenes that produce the same image (denoted Image 2) as Scene 2. This subset is also indicated by a shaded ellipse. A particular scene consistent with Image 2 is indicated by the open circle labeled Scene 2 ˜ . Like Scenes 1 and 1 ˜ , Scenes 2 and 2 ˜ cannot be distinguished from each other by the visual system.

The open ellipse enclosing each solid circle shows a different subset of scenes to which it belongs. These are scenes that share a common illuminant. The open ellipse enclosing Scene 1 indicates all scenes illuminated by E₁(λ), while the open ellipse enclosing Scene 2 indicates all scenes illuminated by E₂(λ).

The figure illustrates why constancy is impossible in general. When viewing Image 1, the visual system cannot tell whether Scene 1 or Scene 1 ˜ is actually present: achromatic points measured for a test patch embedded in these two scenes must be the same, even though the scene illuminants are as different as they are for Scenes 1 and 2. Recall from the data analysis above that this result (no change of achromatic point across a change of illuminant) indicates the absence of constancy.

The figure also illustrates why constancy can be shown across some scene pairs. Scenes 1 and 2 produce distinguishable retinal images, so there is no a priori reason for the measured achromatic points for test patches embedded in these two scenes to bear any relation to each other. In particular, there is no constraint that prevents the change in achromatic points across the two scenes from tracking the corresponding illuminant change. Indeed, one interpretation of the good constancy shown by the data reported above is that the visual system infers approximately the correct illuminants for Scenes 1 and 2. A mystery would occur only if it could also infer the correct illuminants for Scenes 1 ˜ and 2 ˜ .

Figure 61.6 replots the results from achromatic measurements made for Scene 1 together with the results for a new scene, 1 ˜ ˜ . The illuminant in Scene 1 ˜ ˜ is the same as that in Scene 2, but the objects in the scene have been changed to make the image reflected to the eye for Scene 1 ˜ ˜ highly similar to that reflected for Scene 1; Scene 1 ˜ ˜ is an experimental approximation to the idealized Scene 1 ˜ ˜ described above. It would be surprising indeed if constancy were good when assessed between Scenes 1 and 1 ˜ ˜ , and it is not. The achromatic points measured for Scenes 1 and 1 ˜ ˜ are very similar, with the constancy index between them being 0.11.

Figure 61.6..  

Achromatic data when both illuminant and scene surfaces are varied. The images at the top of the figure show the observer's view of two scenes, labeled 1 and 1 ˜ ˜ . The relation between these scenes is described in the text. The test patch is visible in each image. The projection colorimeter was turned off at the time the images were acquired, so the images do not show the results of observers' achromatic adjustments. The chromaticity diagram shows the data from achromatic adjustments of the test patch made in the context of the two scenes. The format is the same as that of Figure 61.4. The equivalent surface reflectance S ˜ (λ) computed from the data obtained in Scene 1 is shown in Figure 61.4. (See color plate 39.)