Rutgers logo

Computational Vision & Psychophysics Lab



Our approach to studying vision and perceptual judgment making begins by posing and answering two questions: (1) what is the optimal computational strategy (the ideal observer) for performing a particular perceptual task given known human limitations, and (2) what additional limitations must be imposed on the optimal computation in order account for measured human performance? We have developed a computational-experimental framework for (a) measuring the information content of complex stimuli within the context of a task, (b) describing the corresponding optimal computation in the form of an ideal observer, (c) investigating potential constraints and predicting probable deviations from optimality, and (d) designing experiments to reveal the underlying computations and uncover the constraints acting on human observers.

The primary focus of our current research program is overt visual search, along with transsaccadic integration and transaccadic memory, which are critical components of overt search. Visual search is an ideal platform for studying perceptual integration and decision making. Search is ubiquitous, occurring as a component of many natural tasks. It is also behaviorally rich, incorporating many important features of more complex visual tasks. Despite this richness, search can be simple enough to allow a tractable formal analysis.

In addition to visual search, our research efforts also include work on perceptual learning and on population coding in visual cortex.

Visual Search and Transsaccadic Integration

Visual Memory for Transsaccadic Integration

Natural visual tasks, such as visual search, typically involve at least several saccadic eye movements, with visual information collected during the intervening fixations. Integrating this visual information across eye movements requires memory. However, very little is known about this transsaccadic memory and how it constrains visual search performance. In the projects described below, we have started to answer these questions. In particular, we measure the limited capacity of transsaccadic memory, we show that this limited capacity can substantially degrade search performance, and we explore how search performance is impacted by various alternative memory allocation strategies. Our lab has begun to investigate this problem.

Intrinsic Position Uncertainty in the Periphery

Efficient performance in visual search and detection requires that observers exclude signals from irrelevant locations. However, phenomena such as crowding and illusory feature conjunctions, as well as evidence from position discrimination studies, suggest that the ability to localize features and thus ignore irrelevant information declines rapidly in the periphery. In the projects described below, we characterize this intrinsic position uncertainty and show that by modeling its effects, we can account for various systematic patterns of behavior in both fixed-gaze and multiple-fixation visual searches.

Perisaccadic Perception

While most real-world tasks involve multiple visual fixations, visual performance thresholds are typically measured during stable fixation. Electrophysiological investigations of visual neurons in tasks involving saccadic eye movements have repeatedly shown that these neurons dramatically change their tuning during the perisaccadic interval (near the onset of an eye movement). Though the perceptual consequences of these changes in neural tuning are not well understood, they suggest that perceptual computations made in the interval preceding an eye movement may differ significantly from those made during stable fixation.

Peripheral Sensitivity and Fixation Selection

A critically important factor in visual search tasks is the rapid decline in sensitivity to target signals in the peripheral visual field. When measured human peripheral sensitivity measurements are incorporated into an ideal observer model of visual search, these patterns of peripheral sensitivity account for many aspects of human search performance.

Search for Categorical Targets in Natural Scenes

Contrary to most laboratory searches, real-world searches typically involve a great deal of uncertainty regarding the target object, whose appearance in natural scenes varies greatly due to differences among individual objects (exemplars) within a target category; to differences in pose, lighting, and viewing angles; and due to partial and self occlusions. In addition, the ability to detect and localize objects in real-world scenes is affected by scene clutter, by scene context, and by the manner in which the search target is indicated to the observer.

One of the most iconic and medically important forms of visual search is the search of x-ray and other medical images for abnormalities. The judgments made by expert radiologists in these searches have real-world implications. This makes radiological search a great applied domain within which to study search. Any insights that can improve these judgments, especially for screenings like routine mammography, will be consequential.

Neural Population Codes in Primary Visual Cortex

Functional Significance of Cortical Topography

Mammalian primary visual cortex (V1) is topographically arranged such that neurons that are spatially adjacent tend to be sensitive to similar features and spatial locations. A fundamental question in systems neuroscience is: what is the functional significance of this topographical organization? I have been examining this question in an ongoing collaboration with Eyal Seidemann’s lab at the University of Texas, which uses electrophysiology and various fluorescence imaging techniques to study visual physiology in behaving monkeys. Our key insight is that if input signals to higher cortical areas involve a pooling of responses from V1 neurons within a local neighborhood, then the local topography should systematically influence the information available for perceptual judgments.

Decoding Correlated Population Responses

Investigators debate the extent to which neural populations use high-order statistical dependencies (correlations) among neural responses to represent information about visual stimuli. Characterizing these high-order dependencies can be difficult, and researchers often decode population responses under the assumption that these correlations contain little information.

Perceptual Learning

Perceptual Cue Acquisition

Learning Complex Feature Spaces

Consider a radiologist who learns to detect lesions or tumors in images generated using a particular imaging method and must later detect similar targets in images generated using a different imaging method. The structure of the noise can change across the two types of images such that different sets of features become diagnostic.