Introduction

Many of the chapters in this handbook discuss what has been learned about how interactions among the different senses affect perception and overt behavior. Although these chapters often use different combinations of sensory modalities (e.g., visual and auditory versus gustatory and olfactory) and different model species to explore multisensory integration, most touch on the same basic observation: cross-modal stimuli activate neural mechanisms that could not be predicted by thinking of the senses as totally independent of one another. Indeed, there is a growing awareness that the brain normally engages in an ongoing synthesis of streams of information that are being carried over the different sensory channels.

The fundamental nature of this property of the vertebrate brain and the potent effects it has on perception and behavior have been the subjects of a rich history of observation, introspection, and speculation, beginning with the earliest philosophers of science. This long interest in multisensory phenomena, coupled with a comparatively recent upsurge in the number of scientific studies exploring them, has resulted in compelling evidence that the information processed by each of the sensory modalities—visual, auditory, somatosensory, vestibular, olfactory, and gustatory—is highly susceptible to influences from the other senses. Even some of the seemingly exotic sensory systems of species less often examined in this context, such as the infrared system of pit vipers that is used for detecting other animals' heat signatures (Hartline, Kass, & Loop, 1978), the chemical and vibratory sensors in the crayfish claw that the animal uses for identifying and obtaining food (Hatt & Bauer, 1980), or the pheromone inputs to the flight motor neurons that are used by moths to locate mates (Olberg & Willis, 1990; Fig. 15.1), share this susceptibility. The pooling of information across sensory modalities appears to be a primordial organizational scheme (see Stein & Meredith, 1993) that has been retained even as organisms have become progressively more complex.

Figure 15.1.  

Multisensory integration in a moth interneuron. This neuron responded to a moving visual stimulus (V, top) and to a pheromone (P, middle). The oscillogram at the top illustrates that the movement of a visual stimulus in one direction elicited more impulses than movement in the other. The pheromone elicited a long train of impulses. However, when the two stimuli were combined, the neuron's directional tuning was significantly enhanced, and the long continuous discharge train elicited by the pheromone was no longer apparent. (Reproduced with permission from Stein, B. E., & Meredith, M. A. [1991]. The merging of the senses. Cambridge, MA: MIT Press. Modified from Olberg & Willis, 1990.)

Despite the considerable vigor and breadth of scientific efforts to detail and understand multisensory integration, however, interactions have not yet been demonstrated among every possible combination of senses: not in studies of human subjects, in which perception is most frequently of primary interest, and certainly not in studies among nonhuman species, where most physiological studies have been conducted and where there is a rich diversity and specialization of sensory systems. Thus, it would be premature to state that multisensory integration is characteristic of all sensory modalities or that it takes place among every combination of them. Nevertheless, until it is shown otherwise, it appears to be a reasonable assumption.

Perhaps as impressive as the ubiquitous nature of multisensory integration is its potency. When multiple sensory cues are provided by the same event, they are usually in close temporal and spatial proximity, and their integration can be of substantial value in enhancing the detection, identification, and orientation to that event (see, e.g., Jiang, Jiang, & Stein, 2000; Stein, Meredith, Huneycutt, & McDade, 1989; Wilkinson, Meredith, & Stein, 1996). They also substantially increase the speed of reactions to these events (see Bernstein, Clark, & Edelstein, 1969; Corneil & Munoz, 1996; Engelken & Stevens, 1989; Frens, Van Opstal, & Van der Willigen, 1995; Gielen, Schmidt, & Van den Heuvel, 1983; Goldring, Dorris, Corneil, Ballantyne, & Munoz, 1996; Harrington & Peck, 1998; Hughes, Reuter-Lorenz, Nozawa, & Fendrich, 1994; Lee, Chung, Kim, & Park, 1991; Perrott, Saberi, Brown, & Strybel, 1990; Zahn, Abel, & Dell'Osso, 1978; see also Marks, Chap. 6, this volume, and Van Opstal & Munoz, Chap. 23, this volume).

Nevertheless, the action of the neural mechanisms that integrate information from multiple sensory systems generally goes unnoticed by the perceiver. There is, after all, nothing particularly noteworthy about the experience. It evokes no signature sensation and is quite commonplace. However, when there are slight discrepancies in the timing or location of cross-modal cues that appear to be derived from the same event, the result is often a cross-modal illusion. Many such illusions have been documented, and many are quite compelling (see, e.g., Shams, Kamitani, & Shimojo, Chap. 2, this volume). These perceptual experiences are not commonplace and do cause one to take notice. They are sometimes amusing, as when a skilled ventriloquist gives the impression that a dummy is speaking (see Woods & Recanzone, Chap. 3, this volume). But at other times they can have a decidedly negative component. What first comes to mind is the effect of watching a film that has been shot from the pilot's seat of a small plane flying rapidly over mountains and into canyons. The visual experience results in vestibular disruptions that, in turn, cause gastrointestinal changes that only patrons of amusement park rides could find amusing. The results of this relatively harmless experience pale in comparison with those that could result from the visual-vestibular illusions that pilots experience and to which they must accommodate in order to avoid making potentially disastrous decisions during take-off and other movements that significantly affect the vestibular and proprioceptive systems (see Lackner & DiZio, Chap. 25, this volume).

What should by evident from the foregoing chapters in this handbook is that sensory events are never experienced in isolation, and thus there is always the opportunity to modify the signals initiated by external stimuli and the perceptions to which they give rise. A unimodal (i.e., modality-specific) stimulus in the external environment normally initiates distinct signals in those receptors that are tuned to it, but these signals are no longer immutable when they reach the brain. They may be incorporated into the stream of ongoing neural activity that is subject to modification from a variety of intrinsic sources, some of which result from activation of interoceptors and others of which are not sensory in the traditional use of the term (e.g., attention and expectation); moreover, these modality-specific signals can be coupled with inputs from other exteroceptors. The particular mix of this central stream of signals can change from moment to moment. Thus, there is little evidence to support the common conviction that, for example, the visual percept of an object is invariant as long as the visual environment is unchanged. On the other hand, there is at least one component of the sensory experience that is likely to be immutable, or at least highly resistant to the influences of other sensory stimuli, and that is its subjective impression. For each sensory modality has evolved a unique subjective impression, or quale. Hue is specific to the visual system, tickle and itch to the somatosensory system, pitch to the auditory systems, and so on. As a result, whether the visual cortex is activated by a natural visual stimulus, by mechanical distortion of the eye, or by direct electrical stimulation of visual centers in the brain itself, the resulting sensation is visual, one whose essential quality is unambiguous.

The lack of ambiguity may seem paradoxical. For if the brain integrates information from different sensory modalities, how can any sensory stimulus give rise to a unique subjective impression? The answer may lie in the dual nature with which the brain represents and deals with sensory information. Many areas of the brain have become specialized to process information on a sense-by-sense basis. These areas do not receive direct projections from more than a single sensory modality, or, if they do, such seemingly “inappropriate” inputs are comparatively few in number. Most prominent among such areas are the nuclei along the primary sensory projection pathways, such as the projection from the retina to the thalamic nucleus for vision (the lateral geniculate nucleus) and from there to primary visual cortex (V1), or the projection from the skin through its relay nucleus in the spinal cord to the thalamic nucleus for somesthesis (the ventrobasal complex) and from there to primary somatosensory cortex (S1). The primary sensory nuclei are recipients of heavy inputs from the peripheral sensory organs and are among the best-studied areas of the central nervous system (CNS). Direct electrical stimulation of the primary sensory cortices has been conducted in awake patients during neurosurgical procedures and has been found to produce modality-specific sensations appropriate for each area (Brindley & Lewin, 1968; Cushing, 1909; Dobelle & Mladejovsky, 1974; Penfield & Boldrey, 1937; Penfield & Rasmussen, 1952). It is also likely that, when activated by natural environmental stimuli, these areas are the sources of modality-specific subjective impressions. Although the prevailing view is that these regions are unaffected by other sensory inputs, a growing body of evidence indicates that their exclusivity may have been overestimated and that even these areas of the nervous system can be affected by information from other sensory modalities (see, in this volume, Schroeder & Foxe, Chap. 18; Cohen & Anderson, Chap. 29; Calvert & Lewis, Chap. 30; Fort & Giard, Chap. 31). Just how functionally significant this cross-modal influence normally is and how this “nonappropriate” information is relayed to these regions are issues of active exploration.

In contrast to presumptive modality-specific regions, many nuclei outside the primary projection pathways are known to receive substantial inputs from multiple sensory modalities. These areas contain mixtures of neurons, some of which are targeted by inputs from only a single sensory modality but many of which receive converging inputs from multiple modalities and thus are rendered multisensory. In some ways multisensory neurons are the functional polar opposites of their modality-specific counterparts wherever they are found, for they are specialized for pooling rather than segregating modality-specific information and probably have no significant role in producing the qualia mentioned earlier. Multisensory areas are found at various levels in the neuraxis and are involved in many different behavioral, perceptual, and emotive processes. One of the best-known sites of multisensory convergence is in a midbrain structure, the superior colliculus (SC), a structure that has served as a model for understanding multisensory integration from anatomical, physiological, and behavioral perspectives.