Skip to main content

Advertisement

SpringerLink
Log in

A neuron-inspired computational architecture for spatiotemporal visual processing

Real-time visual sensory integration for humanoid robots

  • Prospects
  • Published:
Biological Cybernetics Aims and scope Submit manuscript

Abstract

In this article, we present a neurologically motivated computational architecture for visual information processing. The computational architecture’s focus lies in multiple strategies: hierarchical processing, parallel and concurrent processing, and modularity. The architecture is modular and expandable in both hardware and software, so that it can also cope with multisensory integrations – making it an ideal tool for validating and applying computational neuroscience models in real time under real-world conditions. We apply our architecture in real time to validate a long-standing biologically inspired visual object recognition model, HMAX. In this context, the overall aim is to supply a humanoid robot with the ability to perceive and understand its environment with a focus on the active aspect of real-time spatiotemporal visual processing. We show that our approach is capable of simulating information processing in the visual cortex in real time and that our entropy-adaptive modification of HMAX has a higher efficiency and classification performance than the standard model (up to \(\sim \!+6\,\% \)).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. Transport Control Protocol/Internet Protocol, which involves the use of retransmission in the case of message loss.

  2. User Datagram Protocol, which involves unidirectional transmission.

  3. http://web.ics.ei.tum.de/~andreas/videos/ninca.html.

References

  • Anderson J, Lampl I, Gillespie D, Ferster D (2000) The contribution of noise to contrast invariance of orientation tuning in cat visual cortex. Science 290(5498):1968–1972

    Article  CAS  PubMed  Google Scholar 

  • Bar M (2003) A cortical mechanism for triggering top-down facilitation in visual object recognition. J Cogn Neurosci 15(4):600–609

    Article  PubMed  Google Scholar 

  • Belliveau J, Kennedy D, McKinstry R, Buchbinder B, Weisskoff R, Cohen M, Vevea J, Brady T, Rosen B (1991) Functional mapping of the human visual cortex by magnetic resonance imaging. Science 254(5032):716–719

    Article  CAS  PubMed  Google Scholar 

  • Booth M, Rolls E (1998) View-invariant representations of familiar objects by neurons in the inferior temporal visual cortex. Cereb Cortex 8(6):510–523

    Article  CAS  PubMed  Google Scholar 

  • Borst A, Theunissen FE (1999) Information theory and neural coding. Nat Neurosci 2(11):947–957. doi:10.1038/14731

    Article  CAS  PubMed  Google Scholar 

  • Briggs F, Usrey WM (2011) Corticogeniculate feedback and visual processing in the primate. J Physiol 589(1):33–40. doi:10.1113/jphysiol.2010.193599

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • de Garis H, Shuo C, Goertzel B, Ruiting L (2010) A world survey of artificial brain projects, part I: large-scale brain simulations. Neurocomputing 74(1–3):3–29

    Article  Google Scholar 

  • De Valois R, Albrecht D, Thorell L (1982) Spatial frequency selectivity of cells in macaque visual cortex. Vis Res 22(5):545–559

    Article  PubMed  Google Scholar 

  • Deco Gustavo ETR (2004) A Neurodynamical cortical model of visual attention and invariant object recognition. Vis Res 44(6):621–642

    Article  CAS  PubMed  Google Scholar 

  • DeYoe E, Van Essen D (1988) Concurrent processing streams in monkey visual cortex. Trends Neurosci 11(5):219–226

    Article  CAS  PubMed  Google Scholar 

  • Duhamel J, Bremmer F, BenHamed S, Graf W et al (1997) Spatial invariance of visual receptive fields in parietal cortex neurons. Nature 389(6653):845–848

    Article  CAS  PubMed  Google Scholar 

  • Felleman DJ, Van Essen D (1991) Distributed hierarchical processing in the primate cerebral cortex. Cereb Cortex 1(1):1–47

    Article  CAS  PubMed  Google Scholar 

  • Fenske MJ, Aminoff E, Gronau N (2006) Top-down facilitation of visual object recognition: object-based and context-based contributions. Brain 155:3–21

    Google Scholar 

  • Fitzgerald JD, Sincich LC, Sharpee, TO (2011) Minimal models of Multidimensional Computations. PLoS Comput Biol 7(3):e1001111. doi:10.1371/journal.pcbi.1001111

  • Friston KJ, Dolan RJ (2010) Computational and dynamic models in neuroimaging. NeuroImage 52(3):752–765

    Article  PubMed Central  PubMed  Google Scholar 

  • Goerick C, Wersing H, Mikhailova I, Dunn M (2005) Peripersonal space and object recognition for humanoids. In: 5th IEEE-RAS international conference on humanoid robots, pp 387–392

  • Hubel D, Wiesel T (1962) Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J Physiol 160(1):106

    CAS  PubMed Central  PubMed  Google Scholar 

  • Hubel D, Wiesel T (1968) Receptive fields and functional architecture of monkey striate cortex. J Physiol 195(1):215

    CAS  PubMed Central  PubMed  Google Scholar 

  • Huth AG, Nishimoto S, Vu AT, Gallant JL (2012) A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron 76(6):1210–1224. doi:10.1016/j.neuron.2012.10.014

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • Indiveri G, Linares-Barranco B, Hamilton T, Van Schaik A, Etienne-Cummings R, Delbruck T, Liu S, Dudek P, Häfliger P, Renaud S et al (2011) Neuromorphic silicon neuron circuits. Front Neurosci 5

  • Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. Pattern Analysis and Machine Intelligence, IEEE Transactions on 20(11):1254–1259

    Article  Google Scholar 

  • Jones HE, Andolina IM, Ahmed B, Shipp SD, Clements JT, Grieve KL, Cudeiro J, Salt TE, Sillito AM (2012) Differential feedback modulation of center and surround mechanisms in parvocellular cells in the visual thalamus. J Neurosci 32((45)):15,946–15,951

    Article  CAS  Google Scholar 

  • Kourtzi Z, DiCarlo J et al (2006) Learning and neural plasticity in visual object recognition. Curr Opin Neurobiol 16(2):152–158

    Article  CAS  PubMed  Google Scholar 

  • Lee TS, Mumford D (2003) Hierarchical bayesian inference in the visual cortex. JOSA A 20(7):1434–1448

    Article  PubMed  Google Scholar 

  • Livingstone M, Hubel D (1984) Anatomy and physiology of a color system in the primate visual cortex. J Neurosci 4(1):309–356

    CAS  PubMed  Google Scholar 

  • Lomber S, Meredith M, Kral A (2010) Cross-modal plasticity in specific auditory cortices underlies visual compensations in the deaf. Nat Neurosci 13(11):1421–1427

    Article  CAS  PubMed  Google Scholar 

  • Maass W, Legenstein R, Markram H (2002) A new approach towards vision suggested by biologically realistic neural microcircuit models. In: Heinrich HB, Seong-Whan L, Tomaso AP, Christian W (eds) Biologically motivated computer vision, Springer, Berlin, pp 282–293

  • Markram H (2006) The blue brain project. Nat Rev Neurosci 7(2):153–160

    Article  CAS  PubMed  Google Scholar 

  • Mazzoni A, Brunel N, Cavallari S, Logothetis NK, Panzeri S (2011) Cortical dynamics during naturalistic sensory stimulations: experiments and models. J Physiol-Paris 105(1):2–15

    Article  PubMed  Google Scholar 

  • McClurkin J, Optican L, Richmond B et al (1994) Cortical feedback increases visual information transmitted by monkey parvocellular lateral geniculate nucleus neurons. Vis Neurosci 11:601–601

    Article  CAS  PubMed  Google Scholar 

  • Mishra A, Aloimonos Y, Fah C (2009) Active segmentation with fixation. ICCV

  • Modha DS, Ananthanarayanan R, Esser SK, Ndirango A, Sherbondy AJ, Singh R (2011) Cognitive computing. Commun ACM 54(8):62–71

    Article  Google Scholar 

  • Nassi JJ, Callaway EM (2009) Parallel processing strategies of the primate visual system. Nat Rev Neurosci 10(5):360–372

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • Norheim E, Wyller J, Nordlie E, Einevoll GT (2009) Feedback and feedforward contributions to temporal signal processing in the lateral geniculate nucleus. Frontiers in Neuroscience conference abstract: Neuroinformatics

  • Pinotsis D, Moran RJ, Friston KJ (2012) Dynamic causal modeling with neural fields. Neuroimage 59(2):1261–1274

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • Rachmuth G, Shouval H, Bear M, Poon C (2011) A biophysically-based neuromorphic model of spike rate-and timing-dependent plasticity. Proc Nat Acad Sci 108(49):E1266–E1274

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • Rauss K, Schwartz S, Pourtois G (2011) Top-down effects on early visual processing in humans: a predictive coding framework. Neurosci Biobehav Rev 35(5):1237–1253

    Article  PubMed  Google Scholar 

  • Reinagel P, Reid RC (2000) Temporal coding of visual information in the thalamus. J Neurosci: Off J Soc Neurosci 20(14):5392–5400

    CAS  Google Scholar 

  • Riesenhuber M, Poggio T (1999) Hierarchical models of object recognition in cortex. Nat Neurosci 2(11):1019–1025. doi:10.1038/14819

    Article  CAS  PubMed  Google Scholar 

  • Riesenhuber M, Poggio T (2000) Models of object recognition. Nat Neurosci 3:1199–1204

    Article  CAS  PubMed  Google Scholar 

  • Riesenhuber M, Poggio T et al (1999) Hierarchical models of object recognition in cortex. Nat Neurosci 2:1019–1025

    Article  CAS  PubMed  Google Scholar 

  • Roland P, Gulyás B (1995) Visual memory, visual imagery, and visual recognition of large field patterns by the human brain: Functional anatomy by positron emission tomography. Cereb Cortex 5(1):79–93

    Article  CAS  PubMed  Google Scholar 

  • Serre T, Oliva A, Poggio T (2007) A feedforward architecture accounts for rapid categorization. Proc Natl Acad Sci USA 104(15):6424–6429. doi:10.1073/pnas.0700622104

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • Serre T, Wolf L, Bileschi S, Riesenhuber M, Poggio T (2007) Robust object recognition with cortex-like mechanisms. IEEE Trans Pattern Anal Mach Intell 29(3):411–426. doi:10.1109/TPAMI.2007.56

    Google Scholar 

  • Sharpee T, Rust N, Bialek W (2004) Analyzing neural responses to natural signals: maximally informative dimensions. Neural Comput 16(2):223–250

    Article  PubMed  Google Scholar 

  • Sharpee TO, Sugihara H, Kurgansky AV, Rebrik SP, Stryker MP, Miller KD (2006) Adaptive filtering enhances information transmission in visual cortex. Nature 439(7079):936–942. doi:10.1038/nature04519

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • Sherman S, Spear P (1982) Organization of visual pathways in normal and visually deprived cats. Physiol Rev 62(2):738

    CAS  PubMed  Google Scholar 

  • Sillito A, Cudeiro J, Jones H (2006) always returning: feedback and sensory processing in visual cortex and thalamus. Trends Neurosci 29(6):307–316

    Article  CAS  PubMed  Google Scholar 

  • Tanigawa H, Lu HD, Roe AW (2010) Functional organization for color and orientation in macaque V4. Nat Neurosci pp 1–33

  • Thorpe S, Fabre-Thorpe M (2001) Seeking categories in the brain. Science 291(5502):260–263

    Article  CAS  PubMed  Google Scholar 

  • Thorpe S, Fize D, Marlot C et al (1996) Speed of processing in the human visual system. Nature 381(6582):520–522

    Article  CAS  PubMed  Google Scholar 

  • Tovee MJ (1994) How fast is the speed of thought? Curr Biol 4(12):1125–1127

    Article  CAS  PubMed  Google Scholar 

  • Ude A, Atkeson C, Cheng G (2003) Combining peripheral and foveal humanoid vision to detect, pursue, recognize and act. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, pp 2173–2178 (Citeseer)

  • Ude A, Cheng G (2004) Object recognition on humanoids with foveated vision. In 4th IEEE/RAS international conference on humanoid robots, 2004. 1(Humanoids), pp 885–898 (2004)

  • Ude A, Omrčen D, Cheng G (2008) Making object learning and recognition an active process. Int J Humanoid Rob 05(02):267

    Article  Google Scholar 

  • Ude A, Wyart V, Lin L, Cheng G (2005) Distributed visual attention on a humanoid robot. In: Proceedings of IEEE-RAS international conference on humanoid robots, pp 381–386

  • Van Essen DC, Anderson CH, Felleman DJ, Essen V, David C (1992) Information processing in the primate visual system - An integrated systems perspective. Science 255(5043):419–423

    Article  PubMed  Google Scholar 

  • Van Essen DC, Lewis J, Drury H, Hadjikhani N, Tootell R, Bakircioglu M, Miller M (2001) Mapping visual cortex in monkeys and humans using surface-based atlases. Vision research 41(10–11):1359–1378

    Article  PubMed  Google Scholar 

  • Vinje WE, Gallant JL (2000) Sparse coding and decorrelation in primary visual cortex during natural vision. Science (New York, N.Y.) 287(5456):1273–1276

    Article  CAS  Google Scholar 

Download references

Acknowledgments

This work was supported (in part) by the DFG cluster of excellence Cognition for Technical Systems (CoTeSys) of Germany and (in part) by BMBF through the Bernstein Center for Computational Neuroscience Munich (BCCN-Munich).

Author information

Authors and Affiliations

  1. Intstitute for Cognitive Systems (ICS), Technische Universität München, Munich, Germany

    Andreas Holzbach & Gordon Cheng

Authors
  1. Andreas Holzbach
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gordon Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andreas Holzbach.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Holzbach, A., Cheng, G. A neuron-inspired computational architecture for spatiotemporal visual processing. Biol Cybern 108, 249–259 (2014). https://doi.org/10.1007/s00422-014-0597-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00422-014-0597-3

Keywords

Search

Navigation