Skip to main content
SpringerLink
Log in

Extraction of Grasp-Related Visual Features

  • Chapter
  • First Online:
The Visual Neuroscience of Robotic Grasping

Part of the book series: Cognitive Systems Monographs ((COSMOS,volume 28))

  • 1019 Accesses

Abstract

This chapter describes how we model the integration of on-line, action-oriented visual information (dorsal pathway) with knowledge about the target object and memories of previous grasping experiences and object characteristics (ventral pathway). Previous models of vision-based grasping have built so far mainly, when not exclusively, on monkey data. Recent neuropsychological and neuroimaging research has shed a new light on how visuomotor coordination is organized and performed in the human brain. Thanks to such research, a model of vision-based grasping which integrates knowledge coming from single-cell monkey studies with human data can be developed. The basic framework of the proposed model is outlined in this chapter. Final goal of the proposal is to mimic, in a robotic setup, the coordination between sensory, associative and motor cortex of the human brain in vision-based grasping actions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Adams DL, Zeki S (2001) Functional organization of macaque V3 for stereoscopic depth. J Neurophysiol 86(5):2195–2203

    Google Scholar 

  • Anzai A, Chowdhury SA, DeAngelis GC (2011) Coding of stereoscopic depth information in visual areas v3 and v3a. J Neurosci 31(28):10270–10282. doi:10.1523/JNEUROSCI.5956-10.2011

    Google Scholar 

  • Backus BT, Banks MS, van Ee R, Crowell JA (1999) Horizontal and vertical disparity, eye position, and stereoscopic slant perception. Vision Res 39(6):1143–1170

    Article  Google Scholar 

  • Backus BT, Fleet DJ, Parker AJ, Heeger DJ (2001) Human cortical activity correlates with stereoscopic depth perception. J Neurophysiol 86(4):2054–2068

    Google Scholar 

  • Backus BT, Banks MS (1999) Estimator reliability and distance scaling in stereoscopic slant perception. Perception 28(2):217–242

    Article  Google Scholar 

  • Banks MS, Hooge IT, Backus BT (2001) Perceiving slant about a horizontal axis from stereopsis. J Vis 1(2):55–79. 10:1167/1.2.1

    Google Scholar 

  • Bar M, Tootell RB, Schacter DL, Greve DN, Fischl B, Mendola JD, Rosen BR, Dale AM (2001) Cortical mechanisms specific to explicit visual object recognition. Neuron 29(2):529–535

    Article  Google Scholar 

  • Blanz V, Tarr MJ, Bülthoff HH (1999) What object attributes determine canonical views? Perception 28(5):575–599

    Article  Google Scholar 

  • Bradshaw MF, Elliott KM, Watt SJ, Hibbard PB, Davies IRL, Simpson PJ (2004) Binocular cues and the control of prehension. Spat Vis 17(1–2):95–110

    Article  Google Scholar 

  • Brautaset RL, Jennings JAM (2005) Distance vergence adaptation is abnormal in subjects with convergence insufficiency. Ophthalmic Physiol Opt 25(3):211–214. doi:10.1111/j.1475-1313.2005.00274.x

    Google Scholar 

  • Bray S, Arnold AEGF, Iaria G, MacQueen G (2013) Structural connectivity of visuotopic intraparietal sulcus. Neuroimage 82:137–145. doi:10.1016/j.neuroimage.2013.05.080

    Google Scholar 

  • Brouwer GJ, van Ee R, Schwarzbach J (2005) Activation in visual cortex correlates with the awareness of stereoscopic depth. J Neurosci 25(45):10403–10413. doi:10.1523/JNEUROSCI.2408-05.2005

    Google Scholar 

  • Bülthoff HH, Edelman SY, Tarr MJ (1995) How are three-dimensional objects represented in the brain? Cereb Cortex 5(3):247–260

    Article  Google Scholar 

  • Chetverikov D, Szabo Z (1999) A simple and efficient algorithm for detection of high curvature points in planar curves. In: Workshop of the austrian pattern recognition group, pp 175–184

    Google Scholar 

  • Clark JJ, Yuille AL (1990) Data fusion for sensory information processing systems. Springer, Berlin

    Google Scholar 

  • Cumming BG, DeAngelis GC (2001) The physiology of stereopsis. Annu Rev Neurosci 24:203–238. doi:10.1146/annurev.neuro.24.1.203

    Google Scholar 

  • Deneve S, Pouget A (2003) Basis functions for object-centered representations. Neuron 37(2):347–359

    Article  Google Scholar 

  • Dobbins AC, Jeo RM, Fiser J, Allman JM (1998) Distance modulation of neural activity in the visual cortex. Science 281(5376):552–555

    Article  Google Scholar 

  • Einhäuser W, Hipp J, Eggert J, Körner E, König P (2005) Learning viewpoint invariant object representations using a temporal coherence principle. Biol Cybern 93(1):79–90. doi:10.1007/s00422-005-0585-8

    Google Scholar 

  • Ekvall S, Hoffmann F, Kragic D (2003) Object recognition and pose estimation for robotic manipulation using color cooccurrence histograms. In: IEEE International conference on intelligent robots and systems, pp 1284–1289

    Google Scholar 

  • Endo K, Haranaka Y, Shein WN, Adams DL, Kusunoki M, Sakata H (2000) Effects of different types of disparity cues on the response of axis-orientation selective cells in the monkey parietal cortex. Nippon Ganka Gakkai Zasshi 104(5):334–343

    Google Scholar 

  • Ferrier N (1999) Determining surface orientation from fixated eye position and angular visual extent. In: IEEE international conference on robotics and automation, pp 938–943. doi:10.1109/ROBOT.1999.772426

  • Gehrig S, Badino H, Paysan P (2006) Accurate and model-free pose estimation of small objects for crash video analysis. In: British machine vision conference, Edinburgh

    Google Scholar 

  • Genovesio A, Ferraina S (2004) Integration of retinal disparity and fixation-distance related signals toward an egocentric coding of distance in the posterior parietal cortex of primates. J Neurophysiol 91(6):2670–2684. doi:10.1152/jn.00712.2003

    Google Scholar 

  • Georgieva S, Peeters R, Kolster H, Todd JT, Orban GA (2009) The processing of three-dimensional shape from disparity in the human brain. J Neurosci 29(3):727–742. doi:10.1523/JNEUROSCI.4753-08.2009

    Google Scholar 

  • Germann M, Breitenstein MD, Park IK, Pfister H (2007) Automatic pose estimation for range images on the GPU. In: International conference on 3-d digital imaging and modeling, pp 81–90. doi:10.1109/3DIM.2007.13

  • Goddard J (1998) Pose and motion estimation using dual quaternion-based extended Kalman filtering. In: SPIE: three-dimensional image capture and applications, vol 3313

    Google Scholar 

  • Gonzalez F, Perez R (1998) Neural mechanisms underlying stereoscopic vision. Prog Neurobiol 55(3):191–224

    Article  MathSciNet  Google Scholar 

  • Greenwald HS, Knill DC, Saunders JA (2005) Integrating visual cues for motor control: a matter of time. Vis Res 45(15):1975–1989. doi:10.1016/j.visres.2005.01.025

    Google Scholar 

  • Grill-Spector K, Kanwisher N (2005) Visual recognition: as soon as you know it is there, you know what it is. Psychol Sci 16(2):152–160. doi:10.1111/j.0956-7976.2005.00796.x

    Google Scholar 

  • Grill-Spector K, Kushnir T, Hendler T, Edelman S, Itzchak Y, Malach R (1998) A sequence of object-processing stages revealed by fMRI in the human occipital lobe. Hum Brain Mapp 6(4):316–328

    Google Scholar 

  • Heeley DW, Scott-Brown KC, Reid G, Maitland F (2003) Interocular orientation disparity and the stereoscopic perception of slanted surfaces. Spat Vis 16(2):183–207

    Article  Google Scholar 

  • Hillis JM, Watt SJ, Landy MS, Banks MS (2004) Slant from texture and disparity cues: optimal cue combination. J Vis 4(12):967–992. 10:1167/4.12.1

    Google Scholar 

  • Howard IP, Rogers BJ (2002) Seeing in depth. I Porteous

    Google Scholar 

  • Jacobs R (2002) What determines visual cue reliability? Trends in Cogn Sci 6(8):345–350

    Article  Google Scholar 

  • James KH, Humphrey GK, Goodale MA (2001) Manipulating and recognizing virtual objects: where the action is. Can J Exp Psychol 55(2):111–120

    Article  Google Scholar 

  • Jones DG, Malik J (1992) Determining three-dimensional shape from orientation and spatial frequency disparities. In: European conference on computer vision, pp 662–669

    Google Scholar 

  • Julesz B (1971) Foundations of cyclopean perception. MIT Press, Cambridge

    Google Scholar 

  • Knill DC (2007) Robust cue integration: a Bayesian model and evidence from cue-conflict studies with stereoscopic and figure cues to slant. J Vis 7(7):5.1-524. doi:10.1167/7.7.5

    Google Scholar 

  • Konen CS, Mruczek REB, Montoya JL, Kastner S (2013) Functional organization of human posterior parietal cortex: grasping- and reaching-related activations relative to topographically organized cortex. J Neurophysiol 109(12):2897–2908. doi:10.1152/jn.00657.2012

    Google Scholar 

  • Kragic D, Christensen HI (2001) Cue integration for visual servoing. IEEE J Rob Autom 17(1):18–27. doi:10.1109/70.917079

    Google Scholar 

  • Landy MS, Maloney LT, Johnston EB, Young M (1995) Measurement and modeling of depth cue combination: in defense of weak fusion. Vision Res 35(3):389–412

    Article  Google Scholar 

  • Lehky SR, Pouget A, Sejnowski TJ (1990) Neural models of binocular depth perception. Cold Spring Harb Symp Quant Biol 55:765–777

    Article  Google Scholar 

  • Lehky SR, Sejnowski TJ (1990) Neural model of stereoacuity and depth interpolation based on a distributed representation of stereo disparity. J Neurosci 10(7):2281–2299

    Google Scholar 

  • Lippiello V, Siciliano B, Villani L (2001) Position and orientation estimation based on Kalman filtering of stereo images. In: IEEE Int Conf Control Appl 702–707. doi:10.1109/CCA.2001.973950

  • Lippiello V, Siciliano B, Villani L (2006) 3D pose estimation for robotic applications based on a multi-camera hybrid visual system. In: IEEE International conference on robotics and automation, pp 2732–2737

    Google Scholar 

  • Loftus A, Servos P, Goodale MA, Mendarozqueta N, Mon-Williams M (2004) When two eyes are better than one in prehension: monocular viewing and end-point variance. Exp Brain Res 158(3):317–327. doi:10.1007/s00221-004-1905-2

  • Marotta JJ, Perrot TS, Nicolle D, Servos P, Goodale MA (1995) Adapting to monocular vision: grasping with one eye. Exp Brain Res 104(1):107–114

    Article  Google Scholar 

  • Marr D (1982) Vision: a computational investigation into the human representation and processing of visual information. Freeman WH

    Google Scholar 

  • Mon-Williams M, Tresilian JR, Roberts A (2000) Vergence provides veridical depth perception from horizontal retinal image disparities. Exp Brain Res 133(3):407–413

    Article  Google Scholar 

  • Mon-Williams M, Tresilian JR (1999) Some recent studies on the extraretinal contribution to distance perception. Perception 28(2):167–181

    Article  Google Scholar 

  • Naganuma T, Nose I, Inoue K, Takemoto A, Katsuyama N, Taira M (2005) Information processing of geometrical features of a surface based on binocular disparity cues: an fMRI study. Neurosci Res 51(2):147–155. doi:10.1016/j.neures.2004.10.009

    Google Scholar 

  • Norman JF, Todd JT, Phillips F (1995) The perception of surface orientation from multiple sources of optical information. Percept Psychophys 57(5):629–636

    Article  Google Scholar 

  • Orban GA, Janssen P, Vogels R (2006) Extracting 3D structure from disparity. Trends Neurosci 29(8):466–473. doi:10.1016/j.tins.2006.06.012

    Google Scholar 

  • O’Reilly RC, Munakata Y (2000) Computational explorations in cognitive neuroscience—understanding the mind by simulating the brain. MIT Press, Cambridge

    Google Scholar 

  • Parker AJ (2004) From binocular disparity to the perception of stereoscopic depth. In: Chalupa LM, Werner JS (eds) The visual neurosciences, MIT Press, Cambridge, MA, chapter 49, pp 779–792

    Google Scholar 

  • Perry CJ, Tahiri A, Fallah M (2014) Feature integration within and across visual streams occurs at different visual processing stages. J Vis 14(2). doi:10.1167/14.2.10

    Google Scholar 

  • Peters G (2004) Efficient pose estimation using view-based object representations. Mach Vis Appl 16(1):59–63

    Article  Google Scholar 

  • Poggio GF, Gonzalez F, Krause F (1988) Stereoscopic mechanisms in monkey visual cortex: binocular correlation and disparity selectivity. J Neurosci 8(12):4531–4550

    Google Scholar 

  • Pouget S, Sejnowski A (1997) Spatial transformations in the parietal cortex using basis functions. J Cogn Neurosci 9(2):222–237

    Article  Google Scholar 

  • Ray B, Ray K (1995) A new split-and-merge technique for polygonal-approximation of chain coded curves. Pattern Recogn Lett 16(2):161–169

    Article  Google Scholar 

  • Read J (2005) Early computational processing in binocular vision and depth perception. Prog Biophys Mol Biol 87(1):77–108. doi:10.1016/j.pbiomolbio.2004.06.005

    Google Scholar 

  • Riesenhuber M, Poggio T (2000) Models of object recognition. Nat Neurosci 3:1199–1204. doi:10.1038/81479

    Google Scholar 

  • Rolls ET, Webb TJ (2014) Finding and recognizing objects in natural scenes: complementary computations in the dorsal and ventral visual systems. Front Comput Neurosci 8:85. doi:10.3389/fncom.2014.00085

  • Rosenhahn B, Perwass C, Sommer G (2004) Pose estimation of 3D free-form contours. Int J Comput Vis 62:267–289. doi:10.1007/s11263-005-4883-3

    Google Scholar 

  • Rosner B (1975) On the detection of many outliers. Technometrics 17(2):221–227

    Google Scholar 

  • Rousseeuw PJ, Leroy AM (1987) Robust regression and outlier detection. Wiley, New York

    Book  MATH  Google Scholar 

  • Rutschmann RM, Greenlee MW (2004) Bold response in dorsal areas varies with relative disparity level. Neuroreport 15(4):615–619

    Article  Google Scholar 

  • Sakata H, Taira M, Kusunoki M, Murata A, Tanaka Y, Tsutsui K (1998) Neural coding of 3D features of objects for hand action in the parietal cortex of the monkey. Philos Trans R Soc B: Biol Sci 353(1373):1363–1373

    Article  Google Scholar 

  • Sakata H, Taira M, Kusunoki M, Murata A, Tsutsui K, Tanaka Y, Shein WN, Miyashita Y (1999) Neural representation of three-dimensional features of manipulation objects with stereopsis. Exp Brain Res 128(1–2):160–169

    Article  Google Scholar 

  • Salinas E, Thier P (2000) Gain modulation: a major computational principle of the central nervous system. Neuron 27(1):15–21

    Article  Google Scholar 

  • Saxena A, Schulte J, Ng AY (2007) Depth estimation using monocular and stereo cues. In: International joint conferences on artificial intelligence, pp 2197–2203

    Google Scholar 

  • Shikata E, Hamzei F, Glauche V, Knab R, Dettmers C, Weiller C, Büchel C (2001) Surface orientation discrimination activates caudal and anterior intraparietal sulcus in humans: an event-related fMRI study. J Neurophysiol 85(3):1309–1314

    Google Scholar 

  • Taira M, Tsutsui KI, Jiang M, Yara K, Sakata H (2000) Parietal neurons represent surface orientation from the gradient of binocular disparity. J Neurophysiol 83(5):3140–3146

    Google Scholar 

  • Taira M, Nose I, Inoue K, Tsutsui K (2001) Cortical areas related to attention to 3D surface structures based on shading: an fMRI study. Neuroimage 14(5):959–966. doi:10.1006/nimg.2001.0895

    Google Scholar 

  • Taylor G, Kleeman L (2003) Fusion of multimodal visual cues for model-based object tracking. In: Australasian conference on robotics and automation, Brisbane, Australia

    Google Scholar 

  • Teh CH, Chin R (1989) On the detection of dominant points on digital curves. IEEE Trans Pattern Anal Mach Intell 11(8):859–872. doi:10.1109/34.31447

    Google Scholar 

  • Thoma V, Henson RN (2011) Object representations in ventral and dorsal visual streams: fmri repetition effects depend on attention and part-whole configuration. Neuroimage 57(2):513–525. doi:10.1016/j.neuroimage.2011.04.035

    Google Scholar 

  • Thomas OM, Cumming BG, Parker AJ (2002) A specialization for relative disparity in V2. Nat Neurosci 5(5):472–478. doi:10.1038/nn837

    Google Scholar 

  • Todd JT (2004) The visual perception of 3D shape. Trends Cogn Sci 8(3):115–121. doi:10.1016/j.tics.2004.01.006

    Google Scholar 

  • Tresilian JR, Mon-Williams M, Kelly BM (1999) Increasing confidence in vergence as a cue to distance. Proc R Soc B: Biol Sci 266(1414):39–44

    Article  Google Scholar 

  • Tresilian JR, Mon-Williams M (2000) Getting the measure of vergence weight in nearness perception. Exp Brain Res 132(3):362–368

    Article  Google Scholar 

  • Trotter Y, Celebrini S, Stricanne B, Thorpe S, Imbert M (1996) Neural processing of stereopsis as a function of viewing distance in primate visual cortical area V1. J Neurophysiol 76(5):2872–2885

    Google Scholar 

  • Trotter Y, Celebrini S, Durand JB (2004) Evidence for implication of primate area V1 in neural 3-D spatial localization processing. J Physiol Paris 98(1–3):125–134. doi:10.1016/j.jphysparis.2004.03.004

    Google Scholar 

  • Trucco E, Verri A (1998) Introductory techniques for 3-D computer vision. Prentice Hall

    Google Scholar 

  • Tsao DY, Vanduffel W, Sasaki Y, Fize D, Knutsen TA, Mandeville JB, Wald LL, Dale AM, Rosen BR, Essen DCV, Livingstone MS, Orban GA, Tootell RBH (2003) Stereopsis activates V3A and caudal intraparietal areas in macaques and humans. Neuron 39(3):555–568

    Article  Google Scholar 

  • Tsutsui KI, Taira M, Sakata H (2005) Neural mechanisms of three-dimensional vision. Neurosci Res 51(3):221–229. doi:10.1016/j.neures.2004.11.006

    Google Scholar 

  • Tsutsui K, Jiang M, Yara K, Sakata H, Taira M (2001) Integration of perspective and disparity cues in surface-orientation-selective neurons of area CIP. J Neurophysiol 86(6):2856–2867

    Google Scholar 

  • Ullman S (1996) High-level vision. Object recognition and visual cognition. MIT Press, Cambridge

    MATH  Google Scholar 

  • van Ee R, Banks MS, Backus BT (1999) An analysis of binocular slant contrast. Perception 28(9):1121–1145

    Article  Google Scholar 

  • von der Heydt R, Zhou H, Friedman HS (2000) Representation of stereoscopic edges in monkey visual cortex. Vision Res 40(15):1955–1967

    Article  Google Scholar 

  • Wandell BA (1995) Foundations of vision. Sinauer Associates

    Google Scholar 

  • Watt SJ, Bradshaw MF (2003) The visual control of reaching and grasping: binocular disparity and motion parallax. J Exp Psychol Hum Percept Perform 29(2):404–415

    Article  Google Scholar 

  • Weigl A, Hohm K, Seitz M (1995) Processing sensor images for grasping disassembly objects with a parallel-jaw gripper. In: TELEMAN Telerobotics conference

    Google Scholar 

  • Weiss K, Wörn H (2004) Tactile sensor system for an anthropomorphic robot hand. In: IEEE International conference on manipulation and grasping, Genova, Italy

    Google Scholar 

  • Welchman AE, Deubelius A, Conrad V, Bülthoff HH, Kourtzi Z (2005) 3D shape perception from combined depth cues in human visual cortex. Nat Neurosci 8(6):820–827. doi:10.1038/nn1461

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computing, University of Leeds, Leeds, UK

    Eris Chinellato

  2. Department of ICC, Jaume I University, Castellón, Spain

    Angel P. del Pobil

Authors
  1. Eris Chinellato
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Angel P. del Pobil
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eris Chinellato .

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Chinellato, E., del Pobil, A.P. (2016). Extraction of Grasp-Related Visual Features. In: The Visual Neuroscience of Robotic Grasping. Cognitive Systems Monographs, vol 28. Springer, Cham. https://doi.org/10.1007/978-3-319-20303-4_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-20303-4_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-20302-7

  • Online ISBN: 978-3-319-20303-4

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation