Introduction

Analysis of Visual Cortex Neurons

The primary visual cortex plays a very important role in vision and visual perception. To begin with, consider the fact that without area V1, all of the many visual cortical areas (which constitute approximately half of the cerebral cortex in the macaque monkey) are deprived of the visual information relayed through the thalamus from the retina. It has been known for more than a century that damage to this area produces almost total blindness. However, area V1 is not merely a relay station between the thalamus and the other cortical regions. On the contrary, V1 clearly transforms the lateral geniculate nucleus input: in comparison to the relatively simple center-surround receptive fields of geniculate cells, the receptive fields of V1 neurons are considerably more complex and, as will be emphasized in this chapter, the cells are considerably more selective for specific visual features. Finally, consider the anatomical size of the region and the complexity of the neural tissue. In the macaque monkey, V1 constitutes approximately 10% of the entire cerebral cortex, and in comparison to all of the other cortical areas, the primary visual cortex has about twice as many neurons per unit volume, with perhaps half a billion neurons per hemisphere. Given all of these facts, analysis of both the structures and the functions of the primary visual cortex stands as an important challenge to visual neuroscientists in their quest to understand vision and visual perception.

Beginning with the work of Hubel and Wiesel (1962), measurements of the responses of V1 neurons (in the form of action potentials) have provided a wealth of information concerning both the potential biophysical and biochemical mechanisms as well as the ultimate visual information processing functions of these neurons. Nonetheless, in spite of this wealth of scientific information that has accumulated over the decades, in many respects we have taken only a very small step toward a complete understanding of how the visual cortex contributes to visual perception.

In an attempt to analyze the structures and the functions of visual cortex neurons, many researchers have used the well-developed conceptual and mathematical techniques of what can be termed a systems analysis. This quantitative approach was introduced to visual neuroscience in principle by Hartline, who was studying the Limulus visual system (e.g., Ratliff et al., 1974); it has since been used to study the retina, lateral geniculate nucleus, and visual cortex of the primate and related species (e.g., De Valois et al., 1982; Enroth-Cugell and Robson, 1966; Movshon et al., 1978; Ohzawa et al., 1985; Shapley and Victor, 1979; for reviews see Carandini et al., 1999; De Valois and De Valois, 1988; Ferster and Miller, 2000; Geisler and Albrecht, 2000; Palmer et al., 1991; Robson, 1975; Shapley and Lennie, 1985).

Over the past several decades, much research has been devoted to describing both the linear and the nonlinear properties of V1 neurons using a systems analysis. Within this framework, one begins by assessing what aspects of the behavior can be accounted for by simple linear equa-tions and what aspects require nonlinear equations. In this chapter, we describe linear and nonlinear response properties that have been measured within V1 neurons. We then discuss these measurements (and others) within the context of functional transformations of the visual information that ultimately produce high degrees of reliable stimulus selectivity. Finally, we consider several models, at different levels of analysis, of the neural operations that can potentially account for the linear and nonlinear behaviors that have been measured.

Stimulus Selectivity: Features, Filters, and Functions

Research over the past several decades indicates that stimulus selectivity plays a fundamental role in the analysis of visual information within the visual systems of humans, primates, and related species. Within the visual cortex, each neuron is quite selective for a specific visual feature. Currently, there is no agreed-upon intuitive name that adequately captures the presumed function associated with this selectivity (e.g., edge detector, line detector, spatial frequency detector, and so forth). Nonetheless, it is possible to summarize and quantify the selectivity by measuring the responses as a function of many different stimulus dimensions that describe visual stimuli: for example, spatial position, spatial orientation, spatial frequency, temporal frequency, direction of motion, contrast, color, and so forth. These stimulus dimensions are relatively easy to manipulate within the laboratory to measure stimulus selectivity in a systematic, quantitative, and replicable fashion. Further, it is possible to develop descriptive mathematical equations that can adequately describe and summarize the measured responses along these various dimensions. Finally, these descriptive equations can be combined with other equations and analyses to assess the performance characteristics of visual cortex neurons within this multidimensional space and to investigate the ultimate functional consequences of the measured stimulus selectivity.

Rather than attempting to characterize the function of visual cortex neurons using simple intuitive visual feature detection (e.g., edge detection), it is possible to conceptualize the function of each neuron, in a somewhat more neutral fashion by thinking of the function as a filtering operation. A neuron in the primary visual cortex only responds to a specific range of values within a complex (and not necessarily intuitive) multidimensional feature space. In so doing, the neuron filters out the overwhelming majority of unique subsets within the total set and only passes (or signals) the presence of a very small and unique subset. It seems reasonable to assume that whatever this particular type of stimulus selectivity might be, it will probably be closely related to the statistics of natural images (Barlow, 1961).

The observation that cortical neurons are selective for a specific subset of possible visual stimuli has important implications for the overall performance capabilities of cortical neurons: because of this stimulus selectivity, the response of each neuron contains specific information about the presence or absence of a particular feature within the visual stimulus that could be used by a subsequent brain mechanism to detect, discriminate, and identify that specific visual feature. For example, the response magnitude could be used to identify, with a high level of confidence, a specific oriented spatial contour, demarcated by a specific color contrast, moving across a particular location in space, at a particular rate, in a particular direction, and so forth.

This high degree of stimulus selectivity at the level of the visual cortex has led to several different hypotheses regarding the ultimate functional significance of the selectivity. One hypothesis is that the selectivity reflects a sparse code that is well matched to the statistics of natural images (Field, 1987; Olshausen and Field, 1987). A second hypothesis is that the selectivity for local image feature/attributes is a critical step toward the goal of object segregation (Geisler and Albrecht, 2000). A third hypothesis is that the selectivity reflects the sequential hierarchical progression toward neurons within higher cortical regions that are selective for real-world objects (Barlow, 1995). As noted some time ago, sequential filtering is functionally equivalent to pattern recognition (Craik, 1966).

Regardless of whether one, or all, of these hypotheses proves to be accurate, it seems clear that the stimulus selectivity of cortical neurons plays an important role in visual information processing. With this observation in mind, the major focus of this chapter will be those linear and nonlinear properties (and mechanisms) that could potentially have a beneficial influence, or a deleterious influence on stimulus selectivity.

Spatiotemporal Filters and Systems Analysis

In an attempt to characterize both the structures and the functions of visual cortex neurons, from the subcellular level to the behavioral level, many neuroscientists have used the well-developed techniques of systems analysis. The basic principles of this analytical approach have been fully described for the physical sciences as well as the life sciences (e.g., Marmarelis and Marmarelis, 1978; Schwarz and Friedland, 1965) and need not be formally described in this chapter. Stated simply, one attempts to identify and characterize the linear as well as the nonlinear properties of a complex system with the goal of developing a quantitative model that can potentially describe the behavior of the system under a wide range of diverse circumstances. Within this framework, visual cortex neurons can be conceptualized as spatiotemporal filters that respond selectively along several different stimulus dimensions.

There are different methodologies that can be used to investigate a physical system of interest: for example, one can used a frequency domain analysis, a space and/or time domain analysis, a white noise domain analysis, and so forth (see Marmarelis and Marmarelis, 1978). All of these methods have been applied to visual cortex neurons. As noted above, this quantitative systems approach was initially introduced to visual neuroscience by Hartline (and colleagues), but over the past three decades many different laboratories have adopted this approach, and as a consequence we have a rich understanding of both the linear and nonlinear properties of visual cortex neurons (for recent reviews of this literature, see Carandini et al., 1999; Ferster and Miller, 2000; Geisler and Albrecht, 2000).

To simplify this chapter, the frequency domain analysis will be the major focus, although other analyses will be discussed when appropriate. In a frequency domain analysis of visual cortex neurons, the visual stimulus is a spatiotemporal sine wave grating pattern, which can be systematically varied along many different stimulus dimensions. These measurements, and the equations that can be used to describe the responses along the various stimulus dimensions, have provided a quantitative description of the stimulus-response characteristics of visual cortex neurons across a wide and diverse set of circumstances.

Linear Systems Analysis Can Reveal Both Linear and Nonlinear Properties

Over the past half century, linear systems analysis has played a major role in the quantitative analysis of the visual system. As will be described in this chapter, there is overwhelming evidence that visual cortex neurons exhibit a variety of nonlinear properties. Although specific mathematical methods have been developed to study nonlinear systems (e.g., Victor and Knight, 1979; Victor et al., 1977; see also Marmarelis and Marmarelis, 1978), we have learned a great deal about the nonlinear properties of visual cortex neurons by applying linear systems techniques and analyzing the deviations from what would be expected from a linear system.

In a linear system, the response to the sum of several inputs is equal to the sum of the responses to each input individually, and as the amplitude of the stimulus increases the response increases proportionately. There are many standard techniques for characterizing a linear system that could be applied (see Schwarz and Friedland, 1965), and have been applied, to characterize visual cortex neurons. For example, one could measure either the spatiotemporal receptive field or the spatiotemporal transfer function (e.g., DeAngelis et al., 1993; Palmer et al., 1991). If a system is linear, then all of the different techniques give equivalent results and the measurements made with any one of the techniques can be used to predict the responses of the system to arbitrary inputs. However, when a system is nonlinear, two different techniques may give different results; in this case, it becomes essential to use several different techniques and then compare what is similar and what is not. For example, in the case of visual cortex neurons (as will be described below), the spatiotemporal receptive field and the spatiotemporal transfer function are not exactly equivalent. Finally, note that oftentimes, a specific nonlinear mechanism can only be revealed, isolated, and studied using a specific technique. In the case of visual cortex neurons, different nonlinear properties have been discovered and characterized by using different linear systems techniques.

Temporal Dynamics, Stimulus Selectivity, Neural Performance

During natural viewing, eye movements create a rapid progression of diverse images, and because of this, the spatiotemporal contrast can change rapidly over the course of a few hundred milliseconds (for a comprehensive review see Carpenter, 1991). The average duration of a single fixation (during normal saccadic inspection of a visual scene) is approximately 200 msec. This is an important observation to keep in mind when considering the potential effects of a specific linear or nonlinear mechanism on stimulus selectivity and neural performance. If the temporal dynamics of the mechanism are relatively fast, then the mechanism might be able to influence selectivity and performance during a single fixation, based on the spatiotemporal contrast contained within that single fixation. On the other hand, if the temporal dynamics are relatively slow, then the mechanism will not be able to influence stimulus selectivity during a single fixation, based on the spatiotemporal contrast within that fixation.

Over the past several decades, many different laboratories have measured the responses of V1 neurons using drifting spatial frequency gratings, across a wide array of stimulus dimensions, using stimulus durations that are relatively long, to approximate a steady-state condition. These measurements, along with other measurements, have revealed some of the fundamental linear and nonlinear properties of V1 neurons (e.g., De Valois et al., 1982; Movshon et al., 1978; for recent reviews see Carandini et al., 1999; Ferster and Miller, 2000; Geisler and Albrecht, 2000).

The drifting steady-state measurements can be supplemented by measurements of the responses to transient stationary stimuli, where the stimulus durations approximate the fixation durations during natural viewing. Consider using a stationary grating that is presented for a brief interval (200 msec) to measure the responses as a function of some stimulus dimension of interest (e.g., contrast). The measured poststimulus time histograms offer a unique opportunity to examine the temporal dynamics of specific linear and nonlinear properties on a fine time scale. With such a set of measurements, one can ask a wide range of different experimental questions and compare the results of the experiments to what we have learned from the steady-state experiments. Consider the following, somewhat overlapping, subset of possible questions:

Recently, several different laboratories have measured the responses to brief stimuli and analyzed the time course of some of the fundamental properties (e.g., Albrecht et al., 2002; Frazor et al., 1997; Gillespie et al., 2001; Muller et al., 2001; Ringach et al., 1997). Within this chapter, we will consider measurements using drifting steady-state stimuli and measurements using stationary transient stimuli. The results of both types of measurements will be discussed within the context of the effects of the various nonlinearities on stimulus selectivity within a time frame that is comparable to natural viewing.