Basic concepts and formulas

The purpose of deriving an ideal observer is to determine the optimal performance in a task, given the physical properties of the environment and stimuli. Organisms generally do not perform optimally, and hence one should not think of an ideal observer as a potentially realistic model of the actual performance of the organism. Rather, the value of an ideal observer is to provide a precise measure of the stimulus information available for performing the task, a computational theory of how to perform the task, and an appropriate benchmark against which to compare the performance of the organism (Green and Swets, 1966). In addition, the ideal observer can serve as a useful starting point for developing realistic models (e.g., Schrater and Kersten, 2001). With an appropriate ideal observer in hand, one knows how the task should be performed. Thus, it becomes possible to explore in a principled way what the organism is doing right and what it is doing wrong. This can be done by degrading the ideal observer in a systematic fashion by including, for example, hypothesized sources of internal noise (Barlow, 1977), inefficiencies in central decision processes (Barlow, 1977; Green and Swets, 1966; Pelli, 1990), or known anatomical or physiological factors that would limit performance (Geisler, 1989).

Obviously, an ideal observer analysis is sensible only for objective tasks with well-defined performance goals, such as identifying as accurately as possible the physical category to which an object belongs or estimating some physical property of an object. Ideal observer analysis is neither possible nor sensible for subjective tasks such as judging the apparent hue of a stimulus or judging whether a stereoscopic display appears fused or diplopic.

Bayesian Ideal Observers

Most forms of ideal observer analysis are based on the concepts of Bayesian statistical decision theory. To illustrate the Bayesian approach, consider a categorization task where there are n possible stimulus categories, c₁, c₂..., c_n, and the observer's task on each trial is to identify the category correctly, given the particular stimulus S arriving at the eye.1 If there is substantial stimulus noise or overlapping of categories, then the task will be inherently probabilistic. As might be expected intuitively, performance is maximized on average by computing the probability of each category, given the stimulus, and then choosing the category C that is most probable:2,3

Note that “arg max” is just a shorthand notation for a procedure that finds and then returns the category that has the highest probability, given the stimulus.4 In practice, the probability of a category given the stimulus is often computed by making use of Bayes' formula:

where p(c_i|S) is the posterior probability, p(S|c_i) is the likelihood, and p(c_i) is the prior probability.5 The probability in the denominator, p(S), is a constant that is the same for all the categories and hence plays no role in the optimal decision rule. Furthermore, it is completely determined by the likelihoods and prior probabilities:

Substituting Bayes' formula into equation 1, the optimal response is given by

In other words, one can identify a stimulus with maximum accuracy by combining the prior probability of the different categories and the likelihood of the stimulus given each of the possible categories.

In the laboratory, maximizing accuracy is a common goal defined by the experimental design. For this goal, all errors are equally costly, because all errors have the same effect on the accuracy measure. However, this is rarely the case for natural situations, where the costs and benefits associated with different stimulus-response outcomes have a more complex structure. For example, if the goal is survival, then some errors are more costly than others—mistaking a poisonous snake for a branch is more costly than mistaking a branch for a poisonous snake. Within the framework of Bayesian statistical decision theory, more complex goals are represented with a utility function, u(r, ω), which specifies the cost or benefit associated with making response r when the state of the environment is ω (e.g., Berger, 1985). In this more general case, the optimal decision is to make the response, R, that maximizes the average utility over all the possible states of the environment (see footnote 3):

In this decision rule, p(ω) is the prior probability of a given state of the environment, and p(S|ω) is the stimulus likelihood given a state of the environment. Note that equation 4 is special case of equation 5, where the possible states of the environment are the stimulus categories, c₁, c₂..., c_n, the possible responses are the category names, the benefits for all correct responses are equal, and the costs for all incorrect responses are equal.

Constrained Bayesian Ideal Observers

The class of Bayesian ideal observers considered so far operates directly on the stimulus S that arrives at the eye. However, for many applications, it is useful to incorporate some biological constraints into an ideal observer analysis. For example, if good estimates are available for the optics of the eye, the spatial arrangement of the photoreceptors, and their spectral sensitivities, then these estimates can serve as plausible constraints on an ideal observer. In this case, the ideal observer would show the maximum performance possible in the given task, using the photons caught in the photoreceptors. Such an ideal observer must perform worse than one designed to use the photons arriving at the cornea. To the extent that the constraints are accurate, the difference in performance between the ideal observer at the cornea and the one at the level of photon absorptions would provide a precise measure of the information (relevant to the task) lost in the process of image formation and photon capture. Furthermore, the difference in the performance of the ideal observer at the level of photon capture and the performance of the organism as a whole would provide a precise measure of the information lost in the neural processing subsequent to photon capture (e.g., Geisler, 1989).

Another useful way to use constrained ideal observers is to allow some free parameters in the biological constraints and then determine what parameter values produce the best-performing ideal observer. For example, in an ideal observer at the level of photon absorptions, one can allow the peak wavelengths of the receptors to be free parameters and then determine what peak wavelengths would produce the best-performing ideal observer. This would be a precise way of determining how close an organism's photoreceptors are to the optimum for the given task (Regan et al., 2001). Alternatively, the free parameters might represent the receptive field shapes (the configuration of weights placed on each receptor) for some given number of postreceptor neurons. This would be a precise way of determining how close an organism's receptive field shapes are to the optimum for the given task.

A general class of constrained Bayesian ideal observers can be represented by introducing a constraint function, g_θ(S), which maps (either deterministically or probabilistically) the stimulus S at the cornea into an intermediate signal Z = g_θ(S). For example, S might be a vector representing the number of photons entering the pupil from each pixel on a display screen, and Z might be a vector representing the number of photons absorbed in each photoreceptor; hence g_θ(S) would specify the combined effect of the optics, photoreceptor lattice, and photoreceptor absorption spectra. Alternatively, Z might represent the spike count for each ganglion cell, and g_θ(S) would specify the combined effect of the optics and all retinal processing. Any free parameters, such as the peaks of the photoreceptor absorption spectra or the shapes of the receptive fields, are represented in the constraint function by a parameter vector θ.

For any given parameter vector, the optimal decision rule has the same structure as before:

The only difference is that the stimulus S is replaced by the intermediate signal Z. Applying this optimal decision rule typically requires determining the intermediate-signal likelihood p_θ(Z|ω), by combining the constraint function g_θ(S) with the stimulus likelihood distribution p(S|ω). If there are free parameters, then the optimal parameter vector is given by the following formula (e.g., Geisler and Diehl, 2002):

Using θ_opt in equation 6 gives the decision rule for the best-performing ideal observer over the free-parameter space.

The concepts and basic formulas of Bayesian ideal observer analysis are relatively straightforward. However, in specific applications, it can be very difficult to determine or compute the likelihoods, prior probabilities, utility functions, or sums over possible states of the environment. Indeed, there are many situations for which it is not yet possible to determine the performance of the ideal observer. Nonetheless, the number and range of successes have been growing over the years, and the prospects for continued success are good.