Introduction

Neocortical representation or “mapping” of the sensory world is a key feature of functional brain organization in most species. As exemplified by the large body of work on macaque monkeys, each sensory system is mapped onto an interconnected array of cortical areas, and within each array there appears to be a hierarchical progression from simple to complex processing (Burton & Sinclair, 1996; Felleman & Van Essen, 1991; Rauschecker, Tian, Pons, & Mishkin, 1997). The fact that the different senses sample unique aspects of physical objects should provide the brain with both a richer description of objects and converging evidence about their position, movement, and identity. Subjective experience tells us that the inputs are ultimately combined across modalities. However, our understanding of information processing in the brain is largely couched in unisensory terms. The most detailed explication of brain mechanisms of multisensory processing has come from work on the superior colliculus of cats (Stein & Meredith, 1993), although parallel work on the neocortex in humans and monkeys has progressed rapidly in recent years. In monkey studies, evidence of multisensory convergence has been obtained for numerous regions of the parietal (Duhamel, Colby, & Goldberg, 1998; Hyvarinen & Shelepin, 1979; Mazzoni, Bracewell, Barash, & Andersen, 1996; Schroeder & Foxe, 2002; Seltzer & Pandya, 1980), temporal (Benevento, Fallon, Davis, & Rezak, 1977; Bruce, Desimone, & Gross, 1981; Hikosaka, Iwai, Saito, & Tanaka, 1988; Leinonen, 1980; Leinonen, Hyvarinen, & Sovijarvi, 1980; Schroeder & Foxe, 2002; Schroeder, Lindsley, et al., 2001), and frontal lobes (Benevento et al., 1977; Graziano, Hu, & Gross, 1997; Graziano, Yap, & Gross, 1994; Rizzolatti, Scandolara, Gentilucci, & Camarda, 1981a, 1981b). Based on these findings, it is often assumed that multisensory convergence is deferred until late in processing, occurring only in high-order association cortices specialized for that purpose. However, recent findings in both monkeys and humans provide evidence of multisensory convergence at very early, putatively unisensory stages of sensory processing (Calvert, Brammer, Campbell, Iverson, & David, 1999; Foxe et al., 2000; Giard & Peronet, 1999; Levanen, Jousmaki, & Hari, 1998; Molholm et al., 2002; Schroeder & Foxe, 2002; Schroeder et al., 2001). This chapter addresses three questions about multisensory convergence during early cortical processing in humans and monkeys.

The most basic question is, how widespread is multisensory convergence early in cortical processing? If early multisensory convergence is a fundamental design feature of sensory processing architecture, we would expect it to occur across the major sensory systems. On the other hand, early multisensory convergence may be of most use in one or a few of the systems. We will examine the degree to which early sensory processing stages in the visual, auditory, and somatosensory systems receive converging afferents from one or more of the other sensory systems.

Another basic question is, what are the anatomical mechanisms of convergence? In order for interactions between different sensory inputs to occur, the neuronal projections that carry the inputs into the brain must converge at some point. Multisensory convergence can occur within a single neuron (neuronal convergence) or in adjacent neurons within a single cortical region or interconnected ensemble (areal convergence). In theory, converging sensory inputs can be mediated by feed-forward, feedback, or lateral axonal projections. A firm understanding of multisensory processing requires that we define and elaborate anatomical mechanisms in terms of each of these components.

The final question concerns the temporal dimensions of multisensory processing. Multisensory convergence in the neocortex can depend on the temporal parameters of both external stimulus energy transmission and internal (neural) processing, and these parameters vary radically across different sensory modalities. As an example of the former, if we observe a man hitting a spike with a hammer at 20 meters' distance, the sound is delayed relative to the sight of the event. One reason for this perceptual incongruity is that, relative to the speed of light, the speed of sound (∼1100 ft/s) is quite slow, and thus the initial sensory response in the ear begins almost 60 ms after the initial sensory response in the eye. On the other hand, at a comfortable conversational distance of about 1–2 meters, the auditory and visual stimuli generated by a speaker are perceived as synchronous, because the auditory stimulus lag due to the relatively low speed of sound is reduced to the order of 3 or 4 ms. A hidden temporal factor in multisensory convergence is the internal delay, that is, the time necessary for stimulus transduction and neural conduction up to the cortex. Interestingly, the internal processing delay for auditory stimuli is much less than that for visual stimuli, a fact which may adjust the timing offset for auditory-visual stimuli at moderate distances from the observer. Variations in stimulus energy transmission parameters, along with stimulus-related variations in the temporal parameters of processing in the central sensory pathways, contribute a rich temporal dimension to multisensory convergence. We will examine this dimension from both theoretical and empirical perspectives.