Skip to main content
Log in

The modular modality frame model: continuous body state estimation and plausibility-weighted information fusion

Biological Cybernetics Aims and scope Submit manuscript

Abstract

Humans show admirable capabilities in movement planning and execution. They can perform complex tasks in various contexts, using the available sensory information very effectively. Body models and continuous body state estimations appear necessary to realize such capabilities. We introduce the Modular Modality Frame (MMF) model, which maintains a highly distributed, modularized body model continuously updating, modularized probabilistic body state estimations over time. Modularization is realized with respect to modality frames, that is, sensory modalities in particular frames of reference and with respect to particular body parts. We evaluate MMF performance on a simulated, nine degree of freedom arm in 3D space. The results show that MMF is able to maintain accurate body state estimations despite high sensor and motor noise. Moreover, by comparing the sensory information available in different modality frames, MMF can identify faulty sensory measurements on the fly. In the near future, applications to lightweight robot control should be pursued. Moreover, MMF may be enhanced with neural encodings by introducing neural population codes and learning techniques. Finally, more dexterous goal-directed behavior should be realized by exploiting the available redundant state representations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Andersen RA, Snyder LH, Bradley DC, Xing J (1997) Multimodal representation of space in the posterior parietal cortex and its use in planning movements. Ann Rev Neurosci 20:303–330

    Article  PubMed  CAS  Google Scholar 

  • Beauchamp MS (2005) See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex. Curr Opin Neurobiol 15:145–153

    Article  PubMed  CAS  Google Scholar 

  • Bernier PM, Gauthier GM, Blouin J (2007) Evidence for distinct, differentially adaptable sensorimotor transformations for reaches to visual and proprioceptive targets. J Neurophysiol 98:1815–1819

    Article  PubMed  Google Scholar 

  • Botvinick M, Cohen J et al (1998) Rubber hands ‘feel’ touch that eyes see. Nature 391:756

    Article  PubMed  CAS  Google Scholar 

  • Braitenberg V (1986) Vehicles: experiments in synthetic psychology. The MIT Press, Cambridge

    Google Scholar 

  • Bronstein, IN, Semendjajew, KA, Musiol, G, Mühlig, H (eds) (2001) Taschenbuch der Mathematik, 5th edn. Harri Deutsch, Frankfurt

    Google Scholar 

  • Brooks RA (1990) Elephants don’t play chess. Robot Auton Syst 6:3–15

    Article  Google Scholar 

  • Butz MV, Herbort O, Hoffmann J (2007) Exploiting redundancy for flexible behavior: unsupervised learning in a modular sensorimotor control architecture. Psychol Rev 114:1015–1046

    Article  PubMed  Google Scholar 

  • Butz MV, Shirinov E, Reif KL (2010) Self-organizing sensorimotor maps plus internal motivations yield animal-like behavior. Adapt Behav 18:315–337

    Article  Google Scholar 

  • Calvert, GA, Spence, C, Stein, BE (eds) (2004) The handbook of multisensory processes. The MIT Press, Cambridge

    Google Scholar 

  • Chinellato E, Antonelli M, Grzyb BJ, del Pobil AP (2011) Implicit sensorimotor mapping of the peripersonal space by gazing and reaching. IEEE Trans Auton Mental Dev 3:43–53

    Article  Google Scholar 

  • Chinellato E, Grzyb BJ, del Pobil AP (2012) Pose estimation through cue integration: a neuroscience-inspired approach. IEEE Trans Syst Man Cybern Part B 42:530–538

    Article  Google Scholar 

  • Christensen HI, Hager GD (2008) Sensing and estimation. In: Siciliano B, Khatib O (eds) Springer handbook of robotics. Springer, Berlin, chap 4, pp 87–107

  • Cruse H, Steinkühler U (1993) Solution of the direct and inverse kinematic problems by a common algorithm based on the mean of multiple computations. Biol Cybern 69:341–351

    Article  Google Scholar 

  • de Vignemont F, Majid A, Jola C, Haggard P (2009) Segmenting the body into parts: evidence from biases in tactile perception. Q J Exp Psychol 62:500–512

    Article  Google Scholar 

  • Denève S, Duhamel JR, Pouget A (2007) Optimal sensorimotor integration in recurrent cortical networks: a neural implementation of Kalman filters. J Neurosci 27:5744–5756

    Article  PubMed  Google Scholar 

  • Doya K, Ishii S, Pouget A, Rao RPN (2007) Bayesian brain: probabilistic approaches to neural coding. The MIT Press, Cambridge

    Google Scholar 

  • Durrant-Whyte H, Henderson TC (2008) Multisensor data fusion. In: Siciliano B, Khatib O (eds) Springer handbook of robotics. Springer, Berlin, chap 25, pp 585–610

  • Ehrenfeld S, Butz MV (2011) A modular, redundant, multi-frame of reference representation for kinematic chains. In: IEEE International Conference on Robotics and Automation, pp 141–147

  • Erlhagen W, Schöner G (2002) Dynamic field theory of movement preparation. Psychol Rev 109:545–572

    Article  PubMed  Google Scholar 

  • Gadeyne K, Lefebvre T, Bruyninckx H (2005) Bayesian hybrid model-state estimation applied to simultaneous contact formation recognition and geometrical parameter estimation. Int J Robot Res 24:615–630

    Article  Google Scholar 

  • Gentner R, Classen J (2006) Modular organization of finger movements by the human central nervous system. Neuron 52:731–742

    Article  PubMed  CAS  Google Scholar 

  • Gratal X, Romero J, Kragic D (2011) Virtual visual servoing for real-time robot pose estimation. In: Bittanti S, Cenedese A, Zampieri S (eds) World congress, vol 18, pp 9017–9022

  • Herbort O, Butz MV, Pedersen G (2010) The SURE_REACH model for motor learning and control of a redundant arm: from modeling human behavior to applications in robotics. In: Sigaud O, Peters J (eds) From motor learning to interaction learning in robots. Springer, Berlin, pp 85–106

  • Hoffmann M, Marques H, Arieta A, Sumioka H, Lungarella M, Pfeifer R (2010) Body schema in robotics: a review. IEEE Trans Auton Mental Dev 2:304–324

    Article  Google Scholar 

  • Holst E, von Mittelstaedt H (1950) Das Reafferenzprinzip (Wechselwirkungen zwischen Zentralnervensystem und Peripherie). Naturwissenschaften 37:464–476

    Article  Google Scholar 

  • Kohonen T (2001) Self-organizing maps. 3rd ed. Springer, Berlin

    Book  Google Scholar 

  • Körding KP, Wolpert DM (2004) Bayesian integration in sensorimotor learning. Nature 427:244–247

    Article  PubMed  Google Scholar 

  • Latash ML (2008) Synergy. Oxford University Press, New York

    Book  Google Scholar 

  • Latash, ML, Turvey, MT (eds) (1996) Dexterity and its development. Lawrence Erlbaum Assoc Inc, Mahwah

    Google Scholar 

  • Latash ML, Scholz JP, Schöner G (2007) Toward a new theory of motor synergies. Motor Control 11:276–308

    PubMed  Google Scholar 

  • Longo MR, Haggard P (2010) An implicit body representation underlying human position sense. PNAS 107:11727–11732

    Article  PubMed  CAS  Google Scholar 

  • Makin TR, Holmes NP, Ehrsson HH (2008) On the other hand: dummy hands and peripersonal space. Behav Brain Res 191:1–10

    Article  PubMed  Google Scholar 

  • Maravita A, Spence C, Driver J (2003) Multisensory integration and the body schema: close to hand and within reach. Curr Biol 13:531–539

    Article  Google Scholar 

  • McGuire LM, Sabes PN (2009) Sensory transformations and the use of multiple reference frames for reach planning. Nat Neurosci 12:1056–1061

    Article  PubMed  CAS  Google Scholar 

  • Nguyen-Tuong D, Peters J (2011) Model learning for robot control: a survey. Cogn Process 12:319–340

    Article  PubMed  Google Scholar 

  • Pouget A, Dayan P, Zemel R (2003) Inference and computation with population codes. Ann Rev Neurosci 26:381–410

    Article  PubMed  CAS  Google Scholar 

  • Reimann H, Iossifidis I, Schöner G (2010) Integrating orientation constraints into the attractor dynamics approach for autonomous manipulation. In: International conference on humanoid robots, pp 294–301

  • Reimann H, Iossifidis I, Schoner G (2011) Generating collision free reaching movements for redundant manipulators using dynamical systems. In: 2010 IEEE/RSJ international conference on intelligent robots and systems (IROS 2011), pp 5372–5379

  • Ritter H, Martinetz T, Schulten K (1992) Neural computation and self-organizing maps—an introduction. Addison-Wesley, New York

    Google Scholar 

  • Rosenbaum DA (2010) Human motor control. 2nd ed. Academic Press, San Diego

    Google Scholar 

  • Schaal S, Atkeson CG (1998) Constructive incremental learning from only local information. Neural Comput 10:2047–2084

    Article  PubMed  Google Scholar 

  • Scheinmann V, McCarthy JM (2008) Mechanisms and actuation. In: Siciliano B, Khatib O (eds) Springer handbook of robotics. Springer, Berlin, chap 3, pp 67–86

  • Schilling M (2011) Universally manipulable body models—dual quaternion representations in layered and dynamic MMCs. Auton Robots 30:399–425

    Article  Google Scholar 

  • Schmitz J, Schneider A, Schilling M, Cruse H (2008) No need for a body model: positive velocity feedback for the control of an 18-DOF robot walker. Appl Bionics Biomech 5:135–147

    Article  Google Scholar 

  • Shadmehr R, Krakauer JW (2008) A computational neuroanatomy for motor control. Exp Brain Res 185:359–381

    Article  PubMed  Google Scholar 

  • Shadmehr R, Wise SP (2005) The computational neurobiology of reaching and pointing: a foundation for motor learning. MIT Press, Cambridge

    Google Scholar 

  • Sigaud O, Salaun C, Padois V (2011) On-line regression algorithms for learning mechanical models of robots: a survey. Robotics Auton Syst 59:1115–1129

    Article  Google Scholar 

  • Stalph PO, Butz MV (2011) Learning local linear Jacobians for flexible and adaptive robot arm control. Genetic Programm Evolvable Mach 13:137–157

    Article  Google Scholar 

  • Streri A, Pownall TT, Kingerlee ST (1993) Seeing, reaching, touching: the relations between vision and touch in infancy. The MIT Press, Cambridge

    Google Scholar 

  • Tononi G, Edelman GM, Sporns O (1998) Complexity and coherency: integrating information in the brain. Trends Cogn Sci 2:474–484

    Article  PubMed  CAS  Google Scholar 

  • Toussaint M (2006) A sensorimotor map: modulating lateral interactions for anticipation and planning. Neural Comput 18:1132–1155

    Article  PubMed  Google Scholar 

  • Vijayakumar S, D’Souza A, Schaal S (2005) Incremental online learning in high dimensions. Neural Comput 17:2602–2634

    Article  PubMed  Google Scholar 

  • Wells JP, Hyler-Both DL, Danley TD, Wallace GH (2002) Biomechanics of growth and development in the healthy human infant: a pilot study. J Am Osteopath Assoc 102:313–319

    PubMed  Google Scholar 

  • Wolpert DM, Kawato M (1998) Multiple paired forward and inverse models for motor control. Neural Netw 11:1317–1329

    Article  PubMed  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Martin V. Butz.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ehrenfeld, S., Butz, M.V. The modular modality frame model: continuous body state estimation and plausibility-weighted information fusion. Biol Cybern 107, 61–82 (2013). https://doi.org/10.1007/s00422-012-0526-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00422-012-0526-2

Keywords

Navigation