The problem: pattern analysis

Although seldom recognized by either side, both neuroscientists and engineers are faced with a common problem: the problem of pattern analysis, or how to extract structure contained in complex data. Neuroscientists are interested in understanding how the cortex extracts certain properties of the visual environment—surfaces, objects, textures, motion, and so on—from the data stream coming from the retina. Similarly, engineers are interested in designing algorithms capable of extracting structure contained in images or sound—for example, to identify and label parts within the body from medical imaging data. These problems at their core are one in the same, and progress in one domain will likely lead to new insights in the other.

The key difficulty faced by both sides is that the core principles of pattern analysis are not well understood. No amount of experimentation or technological tinkering alone can overcome this obstacle. Rather, it demands that we devote our efforts to advancing new theories of pattern analysis and to directing experimental efforts toward testing these theories.

In recent years, a theoretical framework for how pattern analysis is done by the visual cortex has begun to emerge. The theory has its roots in ideas proposed more than 40 years ago by Attneave and Barlow, and it has been made more concrete in recent years through a combination of efforts in engineering, mathematics, and computational neuroscience. The essential idea is that the visual cortex contains a probabilistic model of images and that the activities of neurons are representing images in terms of this model. Rather than focusing on what features of “the stimulus” are represented by neurons, the emphasis of this approach is on discovering a good featural description of images of the natural environment, using probabilistic models, and then relating this description to the response properties of visual neurons.

In this chapter, I will focus on recent work that has attempted to understand image representation in area V1 in terms of a probabilistic model of natural scenes. The next section will provide an overview of the probabilistic approach and its relation to theories of redundancy reduction and sparse coding. The third section will then describe how this framework has been used to model the structure of natural images, and the fourth section will discuss the relation between these models and the response properties of V1 neurons. Finally, in the fifth section, I will discuss some of the experimental implications of this framework, alternative theories, and the prospects for extending this approach to higher cortical areas.