Introduction

A fundamental issue to neuroscientists engaged in the study of multisensory phenomena is understanding the neurophysiological mechanisms by which the brain combines and integrates information from different sensory channels to form a unified percept.

This question can be approached in multiple ways. Animal studies over the past two decades have shown the existence of multisensory cells in the deep layers of the superior colliculus (SC), a midbrain structure involved in attentive and orienting behaviors, and in various cortical areas in primates and many other mammals (reviewed in Stein & Meredith, 1993; see also Barth, Goldberg, Brett, & Di, 1995; Bruce, Desimone, & Gross, 1981; Hikosaka, Iwai, Saito, & Tanaka, 1988; Wallace, Wilkinson, & Stein, 1996). In addition, the Stein group has shown that the integrative properties of multisensory neurons depend on a set of neural “principles.” The spatial and temporal principles predict that the firing rate of these neurons increases in a superadditive way when two or more stimuli of different modalities are presented in the same location and at the same time; in contrast, spatially or temporally disparate stimuli trigger inhibitory mechanisms that can suppress the responses to either unimodal cue. The principle of inverse effectiveness stipulates that the magnitude of the interaction inversely depends on the effectiveness of the unimodal inputs (Stein & Meredith, 1993). These integration principles devised at the cell level in the SC have also been found to apply to the attentive and orienting behaviors mediated by this structure (Stein, Huneycutt, & Meredith, 1988), and globally to multisensory neurons in the neocortex (Stein & Wallace, 1996).

It is important to note, however, that the collicular and cortical multisensory neurons appear to belong to separate neural circuits, because they are not directly connected to each other (Wallace, Meredith, & Stein, 1993), and they are sensitive in different ways to the spatial factors, the multisensory neurons in the SC being more strictly dependent on the spatial rule of integration than the cortical neurons (Stein & Wallace, 1996). These features suggest that different integration mechanisms may exist for orienting processes (“where” information), mainly governed by the SC and for higher-level perceptual processes (“what” information), mediated by the cortex (Calvert, Hansen, Iversen, & Brammer, 2001; Hughes, Reuter-Lorenz, Nozawa, & Fendrich, 1994; Stein, London, Wilkinson, & Price, 1996).

Although human research on the neurophysiological bases of multisensory integration is much more recent, the rapidly growing number of neuroimaging studies has already provided an impressive number of results (reviewed in Calvert, 2001). Although the data are difficult to compare because of the different sensitivities of the methods—hemodynamic (positron emission tomography [PET] and functional magnetic resonance imaging [fMRI]) versus electromagnetic (electroencephalographic [EEG] and magnetoencephalographic [MEG]) approaches—and the variability of the experimental paradigms used, they already indicate that the physiological mechanisms of multisensory integration are complex and multiple. For example, they depend on the modalities of the sensory inputs (e.g., Calvert et al., 1999; Foxe et al., 2000; Giard & Peronnet, 1999; Macaluso, Frith, & Driver, 2000; Sams et al., 1991) or on the nature (speech/nonspeech) of the information to be combined (Callan, Callan, Kroos, & Vatikiotis-Bateson, 2001; Calvert et al., 2001; Raij, Uutela, & Hari, 2000). Yet when the principles of integration were considered in these studies, they were found to apply both to the magnitude of the physiological effects (Calvert et al., 2001; Widmann, Schröger, Shimojo, & Munka, 2000) and to related behavioral indices (see, however, Stein et al., 1996).

Within the large categories enumerated above (sensory modalities involved, spatial or nonspatial content of information to be bound, speech or nonspeech signals), however, it is tempting to consider multisensory integration as a particular brain function mediated by a specific chain of cross-modal operations; that is, for a given set of sensory inputs, the combination of the unimodal signals would be achieved by “standard” means through roughly invariant neurophysiological processes (Ettlinger & Wilson, 1990).

In this chapter we show, using event-related potentials (ERPs), that the neural mechanisms of multisensory integration in humans cannot be described in terms of hardwired cross-modal operations depending only on the input category. We argue that these mechanisms instead rely on dynamic neural systems with a spatiotemporal organization that depends on exogenous parameters (experimental conditions and tasks) as well as on endogenous factors (expertise of the subject for the task, attention). In addition, some of the neural operations related to multisensory integration may not follow the enhancement/depression rule that characterizes the integrative properties of multisensory cells, and, more generally, some neural principles of multisensory integration may not apply similarly at the cellular level, at the cortical population level, and at the behavioral level.

To this end, we will first discuss some general properties of ERPs and the particular advantages and limits of their use in investigating the physiological processes underlying multisensory integration. Then we present a set of data from three ERP experiments that used the same physical (nonspeech) stimuli but required a different perceptual task for these stimuli. Finally, we discuss these data, taking advantage of the high temporal resolution of the electrophysiological approach, and more broadly with respect to other neuroimaging findings in humans and to the neural principles of multisensory integration that apply at the single-cell level in animals.