Visual–Tactile Fusion Object Recognition Using Joint Sparse Coding

  • Huaping LiuEmail author
  • Fuchun Sun


Visual and tactile measurements offer complementary properties that make them particularly suitable for fusion. It is helpful for the robust and accurate recognition of objects, which is a necessity in many automation systems. In this chapter, a visual–tactile fusion framework is developed for object recognition tasks. This work uses the multivariate time series model to represent the tactile sequence, and the covariance descriptor to characterize the image. Further, a joint group kernel sparse coding method is designed to tackle the intrinsically weak-pairing problem in visual–tactile data samples. Finally, a visual–tactile dataset is developed, which is composed of 18 household objects for validation. The experimental results show that considering both visual and tactile input is beneficial and the proposed method indeed provides an effective strategy for fusion.


  1. 1.
    Allen, P.K.: Integrating vision and touch for object recognition tasks. Int. J. Robot. Res. 7(6), 15–33 (1988)CrossRefGoogle Scholar
  2. 2.
    Beauchamp, M.S.: See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex. Curr. Opin. Neurobiol. 15(2), 145–153 (2005)CrossRefGoogle Scholar
  3. 3.
    Drimus, A., Kootstra, G., Bilberg, A., Kragic, D.: Design of a flexible tactile sensor for classification of rigid and deformable objects. Robot. Auton. Syst. 62(1), 3–15 (2014)CrossRefGoogle Scholar
  4. 4.
    Ernst, M.O., Banks, M.S.: Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870), 429–433 (2002)CrossRefGoogle Scholar
  5. 5.
    Gao, S., Tsang, I.W., Chia, L.T.: Sparse representation with kernels. IEEE Trans. Image Process. 22(2), 423–434 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Heller, M.A.: Visual and tactual texture perception: intersensory cooperation. Atten. Percept. Psychophys. 31(4), 339–344 (1982)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Kroemer, O., Lampert, C.H., Peters, J.: Learning dynamic tactile sensing with robust vision-based training. IEEE Trans. Robot. 27(3), 545–557 (2011)CrossRefGoogle Scholar
  8. 8.
    Lacey, S., Campbell, C., Sathian, K.: Vision and touch: multiple or multisensory representations of objects? Perception 36(10), 1513–1521 (2007)CrossRefGoogle Scholar
  9. 9.
    Lederman, S.J., Klatzky, R.L.: Multisensory texture perception. Handb. Multisens. Process., 107–122 (2004)Google Scholar
  10. 10.
    Liu, H., Qin, J., Cheng, H., Sun, F.: Robust kernel dictionary learning using a whole sequence convergent algorithm. IJCAI 1(2), 5 (2015)Google Scholar
  11. 11.
    Liu, H., Yu, Y., Sun, F., Gu, J.: Visual-tactile fusion for object recognition. IEEE Trans. Autom. Sci. Eng. 14(2), 996–1008 (2017)CrossRefGoogle Scholar
  12. 12.
    Natale, L., Metta, G., Sandini, G.: Learning haptic representation of objects. In: International Conference on Intelligent Manipulation and Grasping (2004)Google Scholar
  13. 13.
    Nene, S.A., Nayar, S.K., Murase, H., et al.: Columbia Object Image Library (coil-20) (1996)Google Scholar
  14. 14.
    Norman, J.F., Norman, H.F., Clayton, A.M., Lianekhammy, J., Zielke, G.: The visual and haptic perception of natural object shape. Atten. Percept. Psychophys. 66(2), 342–351 (2004)CrossRefGoogle Scholar
  15. 15.
    Shekhar, S., Patel, V.M., Nasrabadi, N.M., Chellappa, R.: Joint sparse representation for robust multimodal biometrics recognition. IEEE Trans. Pattern Anal. Mach. Intell. 36(1), 113–126 (2014)CrossRefGoogle Scholar
  16. 16.
    Shrivastava, A., Patel, V.M., Chellappa, R.: Multiple kernel learning for sparse representation-based classification. IEEE Trans. Image Process. 23(7), 3013–3024 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Sprechmann, P., Ramirez, I., Sapiro, G., Eldar, Y.C.: C-hilasso: a collaborative hierarchical sparse modeling framework. IEEE Trans. Signal Process. 59(9), 4183–4198 (2011)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Wang, D., Zhang, Y., Zhou, W., Zhao, H., Chen, Z.: Collocation accuracy of visuo-haptic system: metrics and calibration. IEEE Trans. Haptics 4(4), 321–326 (2011)CrossRefGoogle Scholar
  19. 19.
    Woods, A.T., Newell, F.N.: Visual, haptic and cross-modal recognition of objects and scenes. J. Physiol. Paris 98(1), 147–159 (2004)CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  1. 1.Department of Computer Science and TechnologyTsinghua UniversityBeijingChina

Personalised recommendations