Skip to main content
SpringerLink
Log in

A saliency-based bottom-up visual attention model for dynamic scenes analysis

  • Original Paper
  • Published:
Biological Cybernetics Aims and scope Submit manuscript

Abstract

This work proposes a model of visual bottom-up attention for dynamic scene analysis. Our work adds motion saliency calculations to a neural network model with realistic temporal dynamics [(e.g., building motion salience on top of De Brecht and Saiki Neural Networks 19:1467–1474, (2006)]. The resulting network elicits strong transient responses to moving objects and reaches stability within a biologically plausible time interval. The responses are statistically different comparing between earlier and later motion neural activity; and between moving and non-moving objects. We demonstrate the network on a number of synthetic and real dynamical movie examples. We show that the model captures the motion saliency asymmetry phenomenon. In addition, the motion salience computation enables sudden-onset moving objects that are less salient in the static scene to rise above others. Finally, we include strong consideration for the neural latencies, the Lyapunov stability, and the neural properties being reproduced by the model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Abbott LF, Varela JA, Sen K, Nelson SB (1997) Synaptic depression and cortical gain control. Science 275:220–224

    Article  PubMed  CAS  Google Scholar 

  • Bergen JR, Julesz B (1983) Parallel versus serial processing in rapid pattern discrimination. Nature 303:696–698

    Article  PubMed  CAS  Google Scholar 

  • Bollman M, Hoischen R, Mertsching B (1997) In: Berlin et al. (ed) Integration of static and dynamic scene features guiding visual attention. Springer, Berlin, pp 483–490

  • Borst A (2000) Models of motion detection. Nature neuroscience 3:1168

    Article  PubMed  CAS  Google Scholar 

  • Burt PJ (1988) Proceedings of the 9th international conference on attention mechanisms for vision in dynamic world. Patt Recog 1:977–987

    Google Scholar 

  • Burt PJ, Adelson EH (1983) The Laplacian pyramid as a compact image code. IEEE Trans Commun 31:532–540

    Article  Google Scholar 

  • Cauller L (1995) Layer I of primary sensory neocortex: where top–down converges upon bottom-up. Behav Brain Res 71:163–170

    Article  PubMed  CAS  Google Scholar 

  • Chen B (2005) Mathematical models of motion detection in the fly’s visual cortex. Dissertation, Texas Tech University, Texas

  • Chen LQ, Xie X, Fan X, Ma WY, Zhang HJ, Zhou HQ (2003) A visual attention model for adapting images on small displays. Multimed Syst 9:1–12

    Article  Google Scholar 

  • Coen-Cagli R, Dayan P, Schwartz O (2012) Cortical surround interactions and perceptual salience via natural scene statistics. PLoS Comput Biol 8(3):e1002405

    Article  PubMed  CAS  Google Scholar 

  • Connor CE, Egeth HE, Yantis S (2004) Visual attention: bottom-up versus top–down. Curr Biol 14:R850–R852

    Article  PubMed  CAS  Google Scholar 

  • Conway BR (2001) Spatial structure of cone inputs to color cells in alert macaque primary visual cortex (V-1). J Neurosci 21:2768–2783

    PubMed  CAS  Google Scholar 

  • Conway BR (2009) Color vision, cones and color-coding in the cortex. The Neuroscientist 15:274–290

    Article  PubMed  Google Scholar 

  • De Brecht M, Saiki J (2006) A neural network implementation of a saliency map model. Neural Networks 19:1467–1474

    Article  PubMed  Google Scholar 

  • Deco G, Rolls ET (2004) A neurodyamical cortical model of visual attention and invariant object recognition. Vis Res 44:621–642

    Article  PubMed  Google Scholar 

  • Desimone R, Duncan J (1995) Neural mechanisms of selective visual attention. Ann Rev Neurosci 18:193–222

    Article  PubMed  CAS  Google Scholar 

  • Desimone R, Ungerleider LG (1989) Neural mechanisms of visual processing in monkeys. Elsevier, New York, pp 267–299

  • EC Funded CAVIAR project/IST 2001 37540 http://homepages.inf.ed.ac.uk/rbf/CAVIAR/. Accessed Jan 2011

  • Engel S, Zhang X, Wandell B (1997) Colour tuning in human visual cortex measured with functional magnetic resonance imaging. Nature 388:68–71

    Article  PubMed  CAS  Google Scholar 

  • Fahy FL, Riches IP, Brown MW (1993) Neuronal activity related to visual recognition memory: long-term memory and the encoding of recency and familiarity information in the primate anterior and medial inferior and rhinal cortex. Exp Brain Res 96:457–472

    Article  PubMed  CAS  Google Scholar 

  • Fix J, Rougier N, Alexandre F (2010) A dynamic neural field approach to the covert and overt deployment of spatial attention. Cogn Comput 3:279–293

    Article  Google Scholar 

  • Gao D, Vasconcelos N (2007) Bottom-up saliency is a discriminant process. Proceedings of the IEEE international conference on computer vision, Rio de Janeiro

  • Gonzalez Andino SL, de Peralta Grave (2012) Coding of saliency by ensemble bursting in the amygdala of primates. Front Behav Neurosci 6(38):1–16

    Google Scholar 

  • Greenspan H, Belongie S, Goodman R, Perona P, Rakshit S, Anderson CH (1994) Overcomplete steerable pyramid filters and rotation invariance. Proc IEEE Comput Vis Patt Recog 1:222–228

    Google Scholar 

  • Hamker FH (2004) A dynamic model of how feature cues guide spatial attention. Vis Res 44:501–521

    Article  PubMed  Google Scholar 

  • Hamker FH (2006) Modeling feature-based attention as an active top–down inference process. BioSystems 86:91–99

    Article  PubMed  Google Scholar 

  • Horowitz TS, Wolfe JM, DiMase JS, Klieger SB (2007) Visual search for type of motion is based on simple motion primitives. Perception 36:1624–1634

    Article  PubMed  Google Scholar 

  • Ibbotson M (2001) Identification of mechanisms underlying motion detection in mammals. Springer, Berlin

    Google Scholar 

  • Itti L (2004) Automatic foveation for video compression using a neurobiological model of visual attention. IEEE Trans Image Proc 13:1304–1318

    Article  Google Scholar 

  • Itti L, Koch C (2000) A saliency-based search mechanism for overt and covert shifts of visual attention. Vis Res 40:1489–1506

    Article  PubMed  CAS  Google Scholar 

  • Itti L, Koch C (2001) Computational modeling of visual attention. Nat Rev Neurosci 2:194–203

    Article  PubMed  CAS  Google Scholar 

  • Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Patt Anal Mach Intel 20:1254–1259

    Article  Google Scholar 

  • Jonides J, Yantis S (1988) Uniqueness of abrupt visual onset in capturing attention. Percept Psychophys 43:346–354

    Article  PubMed  CAS  Google Scholar 

  • Kusunoki M, Gottlieb J, Goldberg ME (2000) The lateral intraparietal area as a salience map: the representation of abrupt onset, stimulus motion, and task relevance. Vis Res 40:1459–1468

    Article  PubMed  CAS  Google Scholar 

  • Li Z (1999) Contextual influences in V1 as a basis for pop out and asymmetry in visual search. Proc Natl Acad Sci 96(18):10530–10535

    Article  PubMed  CAS  Google Scholar 

  • Li Z (2002) A saliency map in primary visual cortex. Trend Cogn Sci 6:9–16

    Article  Google Scholar 

  • Liu T, Sun J, Zheng NN, Tang X, Shum HY (2007) Learning to detect a salient object. Proceedings of IEEE computer society conference on computer and vision pattern recognition, Providence

  • López MT, Fenández-Caballero A, Fernández MA, Mira J, Delgado AE (2006) Motion features to enhance scene segmentation in active visual attention. Patt Recog Lett 27:469–478

    Article  Google Scholar 

  • Mahadevan V, Vasconcelos N (2010) Spatiotemporal saliency in dynamic scenes. IEEE Trans Pattern Anal Mach Intell 32: 171–177

    Google Scholar 

  • Matsuno T, Tomonaga M (2006) Visual search for moving and stationary items in chimpanzees (Pan troglodytes) and humans (Homo sapiens). Behav Brain Res 172:219–232

    Article  PubMed  Google Scholar 

  • Maunsell JHR, Treue S (2006) Feature-based attention in visual cortex. Trends Neurosci 29:317–322

    Article  PubMed  CAS  Google Scholar 

  • Meso AI, Zanker JM (2009) Speed encoding in correlation motion detectors as a consequence of spatial structure. Biological Cybern 100:361–370

    Article  Google Scholar 

  • Mira J, Delgado AE, Lopez MT, Fernandez-Caballero A, Fernandez MA (2006) A conceptual frame with two neural mechanisms to model selective visual attention processes. Neurocomputing 71:704–720

    Article  Google Scholar 

  • Mundhenk TN, Itti L (2005) Computational modeling and exploration of contour integration for visual saliency. Biol Cybern 93:188–212

    Article  PubMed  Google Scholar 

  • Nagy AL, Cone SM (1996) Asymmetries in simple feature searches for color. Vis Res 36:2837–2847

    Article  PubMed  CAS  Google Scholar 

  • Navalpakkam V, Itti L (2002) A goal oriented attention guidance model. Lect Notes Comput Sci 2525:453–461

    Article  Google Scholar 

  • Navalpakkam V, Itti L (2005) Modeling the influence of task on attention. Vision Res. 45:205–231

    Article  PubMed  Google Scholar 

  • Navalpakkam V, Itti L (2006) Modeling the influence of task on attention. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 1:1–8

    Google Scholar 

  • Nothdurft H (2000) Salience from feature contrast: additivity across dimensions. Vis Res 40:1183–1201

    Article  PubMed  CAS  Google Scholar 

  • Oliva A, Torralba A, Castelhano MS, Henderson JM (2003) Top–down control of visual attention in object detection. Proc Int Conf Image Proc 1:253–256

    Google Scholar 

  • Park SJ, An KH, Lee M (2002) Saliency map model with adaptive masking based on independent component analysis. Neurocomputing 49:417–422

    Article  Google Scholar 

  • Peters RJ, Iyer A, Itti L, Koch C (2005) Components of bottom-up gaze allocation in natural images. Vis Res 45:2397–2416

    Article  PubMed  Google Scholar 

  • Pointing Gestures: Video Sequence Database (ICPR Workshop, Cambridge, United Kingdom). http://www.cvmt.dk/. Accessed Jan 2011

  • Pomplun M (2007) Advancing area activation towards a general model of eye movements in visual search. In: Gray WD (ed) Integrated models of cognitive systems. Oxford University Press, New York, pp 120–131

    Chapter  Google Scholar 

  • Ramirez-Villegas JF, Ramirez-Moreno DF (2012) Color coding in the cortex: a modified approach to bottom-up visual attention. Biol Cybern. doi:10.1007/s00422-012-0522-6. Accessed on 28 Sept 2012

  • Rapantzikos K, Tsapatsoulis N, Avrithis Y, Kollias S (2007) Bottom-up spatiotemporal visual attention model for video analysis. Image Proc IET 1:237–248

    Article  Google Scholar 

  • Reynolds JH, Heeger DJ (2009) The normalization model of attention. Neuron 61:168–185

    Article  PubMed  CAS  Google Scholar 

  • Royden CS, Wolfe JM, Klempen N (2001) Visual search asymmetries in motion and optic flow fields. Percept Psychophys 63:436–444

    Article  PubMed  CAS  Google Scholar 

  • Santos A, Mier D, Kirsch P, Meyer-Lindenberg A (2011) Evidence for a general face salience signal in human amygdala. Neuroimage 54:3111–3116

    Article  PubMed  Google Scholar 

  • Schrater PR, Knill DC, Simoncelli EP (2000) Mechanisms of visual motion detection. Nature Neuroscience 3:64–68

    Article  PubMed  CAS  Google Scholar 

  • Sejnowski TJ, Koch C, Churchland PS (1988) Computational neuroscience. Science 241:1299–1306

    Article  PubMed  CAS  Google Scholar 

  • Serre T, Wolf L, Bileschi S, Riesenhuber M, Poggio T (2006) Robust object recognition with cortex-like mechanisms. IEEE Trans Patt Anal Mach Intell 29:1–17

    Google Scholar 

  • Shen K, Martin P (2007) Neuronal activity in superior colliculus signals both stimulus identity and saccade goals during visual conjunction search. J Vis 7(5):15: 1–13

    Google Scholar 

  • Shormaker PA, O’Carroll DC, Straw AD (2001) Implementation of visual motion detection with contrast adaptation. Proc SPIE 4591:316–327

    Article  Google Scholar 

  • Sobel KV, Pickard MD, Acklin WT (2009) Using feature preview to investigate the roles of top-down and bottom-up processing in conjunction search. Acta Psychol 132:22–30

    Article  Google Scholar 

  • Thompson KG, Bichot NP (2005) A visual salience map in the primate frontal eye field. Prog Brain Res 147:251–262

    PubMed  Google Scholar 

  • T’so DY, Gilbert CD (1988) The organization of chromatic and spatial interactions in the primate striate cortex. J Neurosci 8:1712–1727

    Google Scholar 

  • Torralba A, Castelhano MS, Oliva A, Henderson JM (2006) Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. Psychol Rev 113:766–786

    Article  PubMed  Google Scholar 

  • Trapp S, Schroll H, Hamker FH (2012). Open and closed loops: A computational approach to attention and consciousness. Adv Cogn Psychol 8(1):1–8. doi:10.2478/v10053-008-0096-y

  • Treisman A, Sykes M, Gelade G (1977) Selective attention stimulus integration. In: Dornic S (ed) Attention and performance VI. Lawrence Erlbaum Associates, New Jersey, pp 333–361

    Google Scholar 

  • Treisman AM, Gelade G (1980) A feature-integration theory of attention. Cogn Psychol 12:97–136

    Article  PubMed  CAS  Google Scholar 

  • Treisman AM, Gormican S (1988) Feature analysis in early vision: evidence from search asymmetries. Psychol Rev 95:15–48

    Article  PubMed  CAS  Google Scholar 

  • Tsodyks M, Pawelzik K, Markram H (1998) Neural networks with dynamic synapses. Neural Comput 10:821–835

    Article  PubMed  CAS  Google Scholar 

  • Tsotsos JK (2001) Motion uniderstanding: task-directed attention and representation that link perception with action. Int J Comput Vis 45:265–280

    Google Scholar 

  • Tsotsos JK, Liu Y, Matinez-Trujillo JC, Pomplun M, Simine E, Zhou K (2005) Attending to visual motion. Comput Vis Image Underst 100:3–40

    Article  Google Scholar 

  • Walther D, Koch C (2006) Modeling attention to salient proto-objects. Neural Networks 19:1395–1407

    Google Scholar 

  • Wilson HR (2004) Spikes, decisions and actions: the dynamical foundations of neuroscience. Oxford University Press, New York

  • Wilson HR, Cowan JD (1973) A mathematical theory of the functional dynamics of cortical and thalamic nervous system. Kybernetik 13:55–80

    Article  PubMed  CAS  Google Scholar 

  • Wolfe J, Butcher SJ, Lee C, Hyle M (2003) Changing your mind: on the contributions of top-down and bottom-up guidance in visual search for feature singletons. J Exp Psychol Hum Percept Perform 29:483–502

    Article  PubMed  Google Scholar 

  • Wolfe JM (2001) Asymmetries in visual search: an introduction. Percept Psychophys 63:381–389

    Article  PubMed  CAS  Google Scholar 

  • Yantis S (1993) Stimulus-driven attentional capture and attentional control settings. J Exp Psychol Hum Percept Perform 19:676–681

    Article  PubMed  CAS  Google Scholar 

  • Zhaoping L (2006) Theoretical understanding of the early visual processes by data compression and data selection. Network: Comput Neural Syst 17:301–334

Download references

Acknowledgments

The authors of this work acknowledge the support of Ruben Coen-Cagli, Albert Einstein College of Medicine, NY, USA.

Author information

Authors and Affiliations

  1. Computational Neuroscience, Department of Physics, Universidad Autonoma de Occidente, Cali, Colombia

    David F. Ramirez-Moreno & Juan F. Ramirez-Villegas

  2. Dominick P. Purpura Department of Neuroscience, Albert Einstein College of Medicine, Yeshiva University, New York, NY, USA

    Odelia Schwartz

  3. Department of Systems and Computational Biology, Albert Einstein College of Medicine, Yeshiva University, New York, NY, USA

    Odelia Schwartz

Authors
  1. David F. Ramirez-Moreno
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Odelia Schwartz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Juan F. Ramirez-Villegas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David F. Ramirez-Moreno.

Additional information

David F. Ramirez-Moreno and Juan F. Ramirez-Villegas contributed equally to the research reported in this work.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ramirez-Moreno, D.F., Schwartz , O. & Ramirez-Villegas, J.F. A saliency-based bottom-up visual attention model for dynamic scenes analysis. Biol Cybern 107, 141–160 (2013). https://doi.org/10.1007/s00422-012-0542-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00422-012-0542-2

Keywords

Search

Navigation