Introduction

For most people, stimulation within a given sense modality yields a unique and distinctive perceptual experience. For instance, the way a banana looks has a different perceptual quality from the way it smells, tastes, and feels. Even within a single modality, various properties of a stimulus trigger unique sensory experiences. Thus, the visual experience associated with the color of a banana is qualitatively different from the visual experience associated with its shape and surface texture.

Although these distinct perceptual qualities may interact to enhance our knowledge of objects and events in the world, they typically remain clearly distinguishable. For individuals with synesthesia, however, the perceptual experiences elicited by the properties of an object tend to interact in ways that are highly unusual. For such individuals, the taste of a particular food may also elicit a distinct tactile sensation (Cytowic, 1993), or the appearance of a letter or digit may be accompanied by a vivid experience of color (Baron-Cohen, Wyke, & Binnie, 1987; Galton, 1880). Research on synesthesia provides a unique opportunity to examine how the brain represents the sensory and conceptual properties of objects, and how learning and experience shape the neural substrates of these representations.

In this chapter we focus on the most common form of synesthesia, in which letters, words, and digits elicit vivid experiences of color when they are seen or heard. These forms are known as color-graphemic and color-phonemic synesthesia, respectively. Other forms of synesthesia seem to be much more rare (Grossenbacher & Lovelace, 2001) and may even reflect different underlying processes (Rich & Mattingley, 2002). Individuals with color-graphemic or color-phonemic synesthesia typically report experiencing specific colors when they hear or read letters, words, or digits. The colors may be seen as a “mist” or as a colored overlay in space, or as a vivid form of mental imagery “in the mind's eye.” Whether these different types of subjective experience reflect different forms of synesthesia is not known, although some authors have drawn such a distinction (see Dixon, Smilek, Wagar, & Merikle, Chap. 52, this volume).

Although the subjective experiences of synesthetes provide important information about the qualitative aspects of the phenomenon, they are difficult to verify experimentally. Our approach has been to use indirect, quantifiable measures of synesthesia to examine its effects on perceptual and cognitive performance. In the first section of this chapter, we review our recent attempts to measure color-graphemic synesthesia objectively and to determine the nature of the representations that give rise to the unusual perceptual experiences. We have found that synesthetic color experiences arise only after substantial early processing of the inducing stimuli (letters and digits), contrary to recent suggestions that synesthetic colors are triggered early in visual processing, prior to the operation of focused attention and explicit recognition of the inducing form (e.g., Palmeri, Blake, Marois, Flanery, & Whetsell, 2002; Ramachandran & Hubbard, 2001a, 2001b). Based on our findings, we have suggested that color-graphemic synesthesia reflects anomalous binding of color and visual form during later, attentive stages of visual processing (Rich & Mattingley, 2002). In support of this hypothesis, we have also shown that synesthetes' color experiences can be substantially modified under conditions of attentional load, or when multiple graphemic forms compete for attention in rivalrous displays.

In the second section of the chapter, we consider the growing evidence from functional brain imaging regarding the neural correlates of synesthesia. Brain imaging studies in color-phonemic synesthetes have revealed significant activity in visual areas of the brain during auditory presentation of words, thus confirming these individuals' subjective reports of vivid color experiences (Nunn et al., 2002; Paulesu et al., 1995). The behavioral tasks used in these studies, however, limit the extent to which they are able to reveal the neural processes that actually give rise to synesthetic experiences. We conclude by suggesting ways in which these limitations might be overcome, and provide some suggestions for future avenues of research.