documentsn.com

Journal of Vision (2008) 8(7):32, 1–20Zhang et al.system (i.e., what it is designed to optimize) is not specified,and it has many parameters that need to be hand-selected.Several saliency algorithms are based on measuring thecomplexity of a local region (Chauvin et al., 2002; Kadir& Brady, 2001; Renninger et al., 2004; Yamada &Cottrell, 1995). Yamada and Cottrell (1995) measure thevariance of 2D Gabor filter responses across differentorientations. Kadir and Brady (2001) measure the entropyof the local distribution of image intensity. Renninger andcolleagues measure the entropy of local features as ameasure of uncertainty, and the most salient point at anygiven time during their shape learning and matching taskis the one that provides the greatest information gainconditioned on the knowledge obtained during previousfixations (Renninger et al., 2004; Renninger, Verghese, &Coughlan, 2007). All of these saliency-as-variance/entropy models are based on the idea that the entropy ofa feature distribution over a local region measures therichness and diversity of that region (Chauvin et al.,2002), and intuitively a region should be salient if itcontains features with many different orientations andintensities. A common critique of these models is thathighly textured regions are always salient regardless oftheir context. For example, human observers find an eggin a nest highly salient, but local-entropy-based algorithmsfind the nest to be much more salient than the egg (Bruce& Tsotsos, 2006; Gao & Vasconcelos, 2004).Gao and Vasconcelos (2004, 2007) proposed a specificgoal for saliency: classification. That is, a goal of thevisual system is to classify each stimulus as belonging to aclass of interest (or not), and saliency should be assignedto locations that are useful for that task. This was firstused for object detection (Gao & Vasconcelos, 2004),where a set of features are selected to best discriminate theclass of interest (e.g., faces or cars) from all other stimuli,and saliency is defined as the weighted sum of featureresponses for the set of features that are salient for thatclass. This forms a definition that is inherently top-downand goal directed, as saliency is defined for a particularclass. Gao and Vasconcelos (2007) define bottom-upsaliency using the idea that locations are salient if theydiffer greatly from their surroundings. They use differenceof Gaussians (DoG) filters and Gabor filters, measuringthe saliency of a point as the Kullback–Leibler (KL)divergence between the histogram of filter responses at thepoint and the histogram of filter responses in thesurrounding region. This addresses a previously mentioned problem commonly faced by complexity-basedmodels (as well as some other saliency models that uselinear filter responses as features): these models alwaysassign high saliency scores to highly textured areas. In theDiscussion section, we will discuss a way that the SUNmodel could address this problem by using non-linearfeatures that model complex cells or neurons in higherlevels of the visual system.Oliva and colleagues proposed a probabilistic model forvisual search tasks (Oliva et al., 2003; Torralba et al.,Downloaded from jov.arvojournals.org on 06/17/202032006). When searching for a target in an image, theprobability of interest is the joint probability that thetarget is present in the current image, together withthe target’s location (if the target is present), given theobserved features. This can be calculated using Bayes’rule:pðO ¼ 1; LjF; GÞ ¼1pðFjGÞ fflfflffl{zfflfflffl}pðFjO ¼ 1; L; GÞpð LjO ¼ 1; GÞpðO ¼ 1jGÞ;bottom up saliencyas defined byTorralba et al:(1)where O 1 denotes the event that the target is present inthe image, L denotes the location of the target when O 1,F denotes the local features at location L, and G denotesthe global features of the image. The global features Grepresent the scene gist. Their experiments show that thegist of a scene can be quickly determined, and the focus oftheir work largely concerns how this gist affects eyemovements. The first term on the right side of Equation 1is independent of the target and is defined as bottom-upsaliency; Oliva and colleagues approximate this conditional probability distribution using the current image’sstatistics. The remaining terms on the right side ofEquation 1 respectively address the distribution of featuresfor the target, the likely locations for the target, and theprobability of the target’s presence, all conditioned on thescene gist. As we will see in the Bayesian framework forsaliency section, our use of Bayes’ rule to derive saliencyis reminiscent of this approach. However, the probabilityof interest in the work of Oliva and colleagues is whetheror not a target is present anywhere in the test image,whereas the probability we are concerned with is theprobability that a target is present at each point in thevisual field. In addition, Oliva and colleagues condition allof their probabilities on the values of global features.Conditioning on global features/gist affects the meaningof all terms in Equation 1, and justifies their use of currentimage statistics for bottom-up saliency. In contrast, SUNfocuses on the effects of an organism’s prior visualexperience.Bruce and Tsotsos (2006) define bottom-up saliencybased on maximum information sampling. Information, inthis model, is computed as Shannon’s self-information,jlog p(F), where F is a vector of the visual featuresobserved at a point in the image. The distribution of thefeatures is estimated from a neighborhood of the point,which can be as large as the entire image. When theneighborhood of each point is indeed defined as the entireimage of interest, as implemented in (Bruce & Tsotsos,2006), the definition of saliency becomes identical to thebottom-up saliency term in Equation 1 from the work of

Journal of Vision (2008) 8(7):32, 1–20Zhang et al.Oliva and colleagues (Oliva et al., 2003; Torralba et al.,2006). It is worth noting, however, that the feature spacesused in the two models are different. Oliva and colleaguesuse biologically inspired linear filters of different orientations and scales. These filter responses are known tocorrelate with each other; for example, a vertical bar inthe image will activate a filter tuned to vertical bars butwill also activate (to a lesser degree) a filter tuned to45-degree-tilted bars. The joint probability of the entirefeature vector is estimated using multivariate Gaussiandistributions (Oliva et al., 2003) and later multivariategeneralized Gaussian distributions (Torralba et al., 2006).Bruce and Tsotsos (2006), on the other hand, employfeatures that were learned from natural images usingindependent component analysis (ICA). These have beenshown to resemble the receptive fields of neurons inprimary visual cortex (V1), and their responses have thedesirable property of sparsity. Furthermore, the featureslearned are approximately independent, so the jointprobability of the features is just the product of eachfeature’s marginal probability, simplifying the probabilityestimation without making unreasonable independenceassumptions.The Bayesian Surprise theory of Itti and Baldi (2005,2006) applies a similar notion of saliency to video. Underthis theory, organisms form models of their environmentand assign probability distributions over the possiblemodels. Upon the arrival of new data, the distributionover possible models is updated with Bayes’ rule, and theKL divergence between the prior distribution and posterior distribution is measured. The more the new dataforces the distribution to change, the larger the divergence. These KL scores of different distributions overmodels combine to produce a saliency score. Itti andBaldi’s implementation of this theory leads to analgorithm that, like the others described here, definessaliency as a kind of deviation from the features present inthe immediate neighborhood, but extending the notion ofneighborhood to the spatiotemporal realm.Bayesian framework for saliencyWe propose that one goal of the visual system is to findpotential targets that are important for survival, such asfood and predators. To achieve this, the visual systemmust actively estimate the probability of a target at everylocation given the visual features observed. We proposethat this probability, or a monotonic transformation of it,is visual saliency.To formalize this, let z denote a point in the visual field.A point here is loosely defined; in the implementationdescribed in the Implementation section, a point corresponds to a single image pixel. (In other contexts, a pointcould refer other things, such as an object; Zhang et al.,Downloaded from jov.arvojournals.org on 06/17/202042007.) We let the binary random variable C denotewhether or not a point belongs to a target class, let therandom variable L denote the location (i.e., the pixelcoordinates) of a point, and let the random variable Fdenote the visual features of a point. Saliency of a point zis then defined as p(C 1 ª F fz, L lz) where fzrepresents the feature values observed at z, and lrepresents the location (pixel coordinates) of z. Thisprobability can be calculated using Bayes’ rule:sz ¼ pðC ¼ 1jF ¼ fz ; L ¼ lz Þ¼pðF ¼ fz ; L ¼ lz jC ¼ 1ÞpðC ¼ 1Þ:pðF ¼ fz ; L ¼ lz Þð2ÞWe assume1 for simplicity that features and location areindependent and conditionally independent given C 1:pðF ¼ fz ; L ¼ lz Þ ¼ pðF ¼ fz ÞpðL ¼ lz Þ;ð3ÞpðF ¼ fz ; L ¼ lz jC ¼ 1Þ¼ pðF ¼ fz jC ¼ 1ÞpðL ¼ lz jC ¼ 1Þ:ð4ÞThis entails the assumption that the distribution of afeature does not change with location. For example,Equation 3 implies that a point in the left visual field isjust as likely to be green as a point in the right visual field.Furthermore, Equation 4 implies (for instance) that a pointon a target in the left visual field is just as likely to begreen as a point on a target in the right visual field. Withthese independence assumptions, Equation 2 can berewritten as:sz ¼pðF ¼ fz jC ¼ 1ÞpðL ¼ lz jC ¼ 1ÞpðC ¼ 1ÞpðF ¼ fz ÞpðL ¼ lz Þð5Þ¼pðF ¼ fz jC ¼ 1Þ pðL ¼ lz jC ¼ 1ÞpðC ¼ 1ÞpðF ¼ fz ÞpðL ¼ lz Þð6Þ¼1pðF ¼ fz Þ fflfflfflfflffl{zfflfflfflfflffl}pðF ¼ fz jC ¼ 1Þ pðC ¼ 1jL ¼ lz Þ fflfflfflfflfflfflfflffl} fflfflfflfflfflfflfflffl}LikelihoodLocation prior ��fflfflfflfflfflffl}Independentof targetDependent on targetðbottom up saliencyÞðtop down knowledgeÞ. ð7ÞTo compare this probability across locations in animage, it suffices to estimate the log probability (sincelogarithm is a monotonically increasing function). For this

Journal of Vision (2008) 8(7):32, 1–20Zhang et al.reason, we take the liberty of using the term saliency torefer both to sz and to log sz, which is given by:log sz ¼ jlog pðF ¼ fz Þ þ log pðF ¼ fz jC ¼ 1Þ fflfflfflfflfflffl} elf informationpðF ¼ fz ; C ¼ 1Þ¼ log:pðF ¼ fz ÞpðC ¼ 1Þ �fflfflffl}ð11ÞPointwise mutual informationðoverall saliencyÞLog likelihoodþ log pðC ¼ 1jL ¼ lz Þ : �8ÞLocation priorThe first term on the right side of this equation, jlogp(F fz), depends only on the visual features observed atthe point and is independent of any knowledge we haveabout the target class. In information theory, jlog p(F fz)is known as the self-information of the random variableF when it takes the value fz. Self-information increaseswhen the probability of a feature decreasesVin otherwords, rarer features are more informative. We havealready discussed self-information in the context ofprevious work, but as we will see later, SUN’s use ofself-information differs from that of previous approaches.The second term on the right side of Equation 8, logp(F fz ª C 1), is a log-likelihood term that favorsfeature values that are consistent with our knowledge ofthe target. For example, if we know that the target isgreen, then the log-likelihood term will be much larger fora green point than for a blue point. This corresponds to thetop-down effect when searching for a known target,consistent with the finding that human eye movementpatterns during iconic visual search can be accounted forby a maximum likelihood procedure for computing themost likely location of a target (Rao, Zelinsky, Hayhoe, &Ballard, 2002).The third term in Equation 8, log p(C 1 ª L lz), isindependent of visual features and reflects any priorknowledge of where the target is likely to appear. It hasbeen shown that if the observer is given a cue of where thetarget is likely to appear, the observer attends to thatlocation (Posner & Cohen, 1984). For simplicity andfairness of comparison with Bruce and Tsotsos (2006),Gao and Vasconcelos (2007), and Itti et al. (1998), weassume location invariance (no prior information aboutthe locations of potential targets) and omit the locationprior; in the Results section, we will further discuss theeffects of the location prior.After omitting the location prior from Equation 8, theequation for saliency has just two terms, the self-informationand the log-likelihood, which can be combined:log sz ¼ jlog pðF ¼ fz Þ þ log pðF ¼ fz jC ¼ 1Þ fflfflfflfflfflffl} elf informationðbottom up saliencyÞ¼ log5Log likelihoodðtop down knowledgeÞpðF ¼ fz jC ¼ 1ÞpðF ¼ fz ÞDownloaded from jov.arvojournals.org on 06/17/2020ð9Þð10ÞThe resulting expression, which is called the pointwisemutual information between the visual feature and thepresence of a target, is a single term that expresses overallsaliency. Intuitively, it favors feature values that are morelikely in the presence of a target than in a target’s absence.When the organism is not actively searching for aparticular target (the free-viewing condition), the organism’s attention should be directed to any potential targetsin the visual field, despite the fact that the featuresassociated with the target class are unknown. In this case,the log-likelihood term in Equation 8 is unknown, so weomit this term from the calculation of saliency (this canalso be thought of as assuming that for an unspecifiedtarget, the likelihood distribution is uniform over featurevalues). In this case, the overall saliency reduces to justthe self-information term: log sz jlog p(F fz). We takethis to be our definition of bottom-up saliency. It impliesthat the rarer a feature is, the more it will attract ourattention.This use of jlog p(F fz) differs somewhat from how itis often used in the Bayesian framework. Often, the goalof the application of Bayes’ rule when working withimages is to classify the provided image. In that case, thefeatures are given, and the jlog p(F fz) term functionsas a (frequently omitted) normalizing constant. When thetask is to find the point most likely to be part of a targetclass, however, jlog p(F fz) plays a much moresignificant role as its value varies over the points of theimage. In this case, its role in normalizing the likelihood ismore important as it acts to factor in the potentialusefulness of each feature to aid in discrimination.Assuming that targets are relatively rare, a target’s featureis most useful if that feature is comparatively rare in thebackground environment, as otherwise the frequency withwhich that feature appears is likely to be more distractingthan useful. As a simple illustration of this, consider thateven if you know with absolute certainty that the target isred, i.e. p(F red ª C 1) 1, that fact is useless ifeverything else in the world is red as well.When one considers that an organism may be interested ina large number of targets, the usefulness of rare featuresbecomes even more apparent; the value of p(F fz ª C 1)will vary for different target classes, while p(F fz) remainsthe same regardless of the choice of targets. While thereare specific distributions of p(F fz ª C 1) for whichSUN’s bottom-up saliency measure would be unhelpful infinding targets, these are special cases that are not likely tohold in general (particularly in the free-viewing condition,where the set of potential targets is largely unknown).That is, minimizing p(F fz) will generally advance thegoal of increasing the ratio p(F fz ª C 1) / p(F fz),

Journal of Vision (2008) 8(7):32, 1–20Zhang et al.implying that points with rare features should be found“interesting.”Note that all of the probability distributions describedhere should be learned by the visual system throughexperience. Because the goal of the SUN model is to findpotential targets in the surrounding environment, theprobabilities should reflect the natural statistics of theenvironment and the learning history of the organism,rather than just the statistics of the current image. (This isespecially obvious for the top-down terms, which requirelearned knowledge of the targets.)In summary, calculating the probability of a target ateach point in the visual field leads naturally to theestimation of information content. In the free-viewingcondition, when there is no specific target, saliencyreduces to the self-information of a feature. This impliesthat when one’s attention is directed only by bottom-upsaliency, moving one’s eyes to the most salient points inan image can be regarded as maximizing informationsampling, which is consistent with the basic assumption ofBruce and Tsotsos (2006). When a particular target isbeing searched for, on the other hand, our model impliesthat the best features to attend to are those that have themost mutual information with the target. This has beenshown to be very useful in object detection with objectssuch as faces and cars (Ullman, Vidal-Naquet, & Sali,2002).In the rest of this paper, we will concentrate on bottomup saliency for static images. This corresponds to the freeviewing condition, when no particular target is of interest.Although our model was motivated by the goal-orientedtask of finding important targets, the bottom-up component remains task blind and has the statistics of naturalscenes as its sole source of world knowledge. In the nextsection, we provide a simple and efficient algorithm forbottom-up saliency that (as we demonstrate in the Resultssection) produces state-of-the-art performance in predicting human fixations.6Gabor filters (e.g., Itti & Koch, 2001; Itti et al., 1998;Oliva et al., 2003; Torralba et al., 2006). In Bruce andTsotsos (2006), the features are calculated as theresponses to filters learned from natural images usingindependent component analysis (ICA). In this paper, weconduct experiments with both kinds of features.Below, we describe the SUN algorithm for estimatingthe bottom-up saliency that we derived in the Bayesianframework for saliency section, jlog p(F fz). Here, apoint z corresponds to a pixel in the image. For theremainder of the paper, we will drop the subscript z fornotational simplicity. In this algorithm, F is a randomvector of filter responses, F [F1, F2, I], where therandom variable Fi represents the response of the ith filterat a pixel, and f [ f1, f2, I] are the values of these filterresponses at this pixel location.Method 1: Difference of Gaussians filtersAs noted above, many existing models use a collectionof DoG (difference of Gaussians) and/or Gabor filters asthe first step of processing the input images. These filtersare popular due to their resemblance to the receptive fieldsof neurons in the early stages of the visual system, namelythe lateral geniculate nucleus of the thalamus (LGN)and the primary visual cortex (V1). DoGs, for example,give the well-known “Mexican hat” center-surround filter.Here, we apply DoGs to the intensity and color channelsof an image. A more complicated feature set composed ofa mix of DoG and Gabor filters was also initiallyevaluated, but results were similar to those of the simpleDoG filters used h

review, see Findlay & Gilchrist, 2003). Itti and Koch’s saliency model (Itti & Koch, 2000, 2001; Itti et al., 1998) is one of the earliest and the most used for comparison in later work. The model is an implementati