Towards a Theoretical Framework for Learning Multi-modal Patterns for Embodied Agents

  • Nicoletta Noceti
  • Barbara Caputo
  • Claudio Castellini
  • Luca Baldassarre
  • Annalisa Barla
  • Lorenzo Rosasco
  • Francesca Odone
  • Giulio Sandini
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5716)


Multi-modality is a fundamental feature that characterizes biological systems and lets them achieve high robustness in understanding skills while coping with uncertainty. Relatively recent studies showed that multi-modal learning is a potentially effective add-on to artificial systems, allowing the transfer of information from one modality to another. In this paper we propose a general architecture for jointly learning visual and motion patterns: by means of regression theory we model a mapping between the two sensorial modalities improving the performance of artificial perceptive systems. We present promising results on a case study of grasp classification in a controlled setting and discuss future developments.


multi-modality visual and sensor-motor patterns regression theory behavioural model objects and actions recognition 


  1. 1.
    Rizzolatti, G., Craighero, L.: The Mirror-Neuron System. Annual Review of Neuroscience 27, 169–192 (2004)CrossRefGoogle Scholar
  2. 2.
    Harris, C., Stephens, M.: A Combined Corner and Edge Detector. In: Proceedings of The Fourth Alvey Vision Conference, pp. 147–151 (1988)Google Scholar
  3. 3.
    Mikolajczyk, K., Schmid, C.: Scale and Affine Invariant Interest Point Detectors. IJCV 60(1), 63–86 (2004)CrossRefGoogle Scholar
  4. 4.
    Lowe, D.G.: Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision 60(2), 91–110 (2004)CrossRefGoogle Scholar
  5. 5.
    Mikolajczyk, K., Schmid, C.: A Performance Evaluation of Local Descriptors. Trans on PAMI 27(10) (2005)Google Scholar
  6. 6.
    Csurka, G., Dance, C.R., Fan, L., Bray, C.: Visual Categorization with Bag of Keypoints. In: ECCV (2004)Google Scholar
  7. 7.
    Ferrari, V., Tuytelaars, T., Van Gool, L.: Simultaneous Object Recognition and Segmentation from Single or Multiple Model Views. IJVC 67(2) (2006)Google Scholar
  8. 8.
    Lo Gerfo, L., Rosasco, L., Odone, F., De Vito, E., Verri, A.: Spectral Algorithms for Supervised Learning. Neural Computation 20(7) (2008)Google Scholar
  9. 9.
    Yao, Y., Rosasco, L., Caponnetto, A.: On Early Stopping in Gradient Descent Learning. Constructive Approximation 26(2) (2007)Google Scholar
  10. 10.
    Micchelli, C.A., Pontil, M.: On learning vector-valued functions. Neural Computation 17 (2005)Google Scholar
  11. 11.
    De Vito, E., Rosasco, L., Caponnetto, A., Piana, M., Verri, A.: Some Properties of Regularized Kernel Methods. Journal of Machine Learning Research 5 (2004)Google Scholar
  12. 12.
    Baldassarre, L., Barla, A., Rosasco, L., Verri, A.: Learning vector valued functions with spectral regularization (preprint)Google Scholar
  13. 13.
    Gallese, V., Fadiga, L., Fogassi, L., Rizzolatti, G.: Action Recognition in the Premotor Cortex. Brain 119, 593–609 (1996)CrossRefGoogle Scholar
  14. 14.
    Metta, G., Sandini, G., Natale, L., Craighero, L., Fadiga, L.: Understanding Mirror Neurons: A Bio-Robotic Approach. Interaction Studies 7, 197–232 (2006)CrossRefGoogle Scholar
  15. 15.
    Hartigan, J.A., Wong, M.A.: A K-Means Clustering Algorithm. Applied Statistics 28(1) (1979)Google Scholar
  16. 16.
    Cutkosky, M.: On grasp choice, grasp models and the design of hands for manufacturing tasks. IEEE Transactions on Robotics and Automation (1989)Google Scholar
  17. 17.
    Castellini, C., Orabona, F., Metta, G., Sandini, G.: Internal Models of Reaching and Grasping. Advanced Robotics 21(13) (2007)Google Scholar
  18. 18.
    Buhlmann, P.: Boosting for High-Dimensional Linear Models. Annals of Statistics 34(2) (2006)Google Scholar
  19. 19.
    Micchelli, C.A., Pontil, M.: Kernels for Multi-task Learning. In: NIPS (2004)Google Scholar
  20. 20.
    Rifkin, R., Yeo, G., Poggio, T.: Regularized Least-Squares Classification. In: Advances in Learning Theory: Methods, Models and Applications (2003)Google Scholar
  21. 21.
    Argyriou, A., Maurer, A., Pontil, M.: An Algorithm for Transfer Learning in a Heterogeneous Environment. In: ECML/PKDD (1), pp. 71–85 (2008)Google Scholar
  22. 22.
    Jacob, L., Bach, F., Vert, J.P.: Clustered Multi-Task Learning: a Convex Formulation. In: NIPS (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Nicoletta Noceti
    • 1
  • Barbara Caputo
    • 2
  • Claudio Castellini
    • 3
  • Luca Baldassarre
    • 1
    • 4
  • Annalisa Barla
    • 1
  • Lorenzo Rosasco
    • 1
    • 5
  • Francesca Odone
    • 1
  • Giulio Sandini
    • 3
    • 6
  1. 1.DISI - University of GenovaItaly
  2. 2.IDIAP - MartignySwitzerland
  3. 3.DIST - University of GenovaItaly
  4. 4.DIFI - University of GenovaItaly
  5. 5.MITCambridgeUSA
  6. 6.IITGenovaItaly

Personalised recommendations