Journal of Computer Science and Technology

, Volume 30, Issue 2, pp 340–352 | Cite as

RGB-D Hand-Held Object Recognition Based on Heterogeneous Feature Fusion

  • Xiong Lv
  • Shu-Qiang JiangEmail author
  • Luis Herranz
  • Shuang Wang
Regular Paper


Object recognition has many applications in human-machine interaction and multimedia retrieval. However, due to large intra-class variability and inter-class similarity, accurate recognition relying only on RGB data is still a big challenge. Recently, with the emergence of inexpensive RGB-D devices, this challenge can be better addressed by leveraging additional depth information. A very special yet important case of object recognition is hand-held object recognition, as manipulating objects with hands is common and intuitive in human-human and human-machine interactions. In this paper, we study this problem and introduce an effective framework to address it. This framework first detects and segments the hand-held object by exploiting skeleton information combined with depth information. In the object recognition stage, this work exploits heterogeneous features extracted from different modalities and fuses them to improve the recognition accuracy. In particular, we incorporate handcrafted and deep learned features and study several multi-step fusion variants. Experimental evaluations validate the effectiveness of the proposed method.


RGB-D hand-held object recognition heterogeneous features fusion 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Li L, Jiang S, Huang Q. Learning hierarchical semantic description via mixed-norm regularization for image understanding. IEEE Transactions on Multimedia, 2012, 14(5):1401–1413.CrossRefGoogle Scholar
  2. [2]
    Bo L, Ren X, Fox D. Unsupervised feature learning for RGB-D based object recognition. In Springer Tracts in Advanced Robotics 88, Desai J P, Dudek G, Khatib O, Kumar V (eds.), Springer, pp.387–402.Google Scholar
  3. [3]
    Gupta S, Arbeláez P, Girshick R, Malik J. Indoor scene understanding with RGB-D images: Bottom up segmentation, object detection and semantic segmentation. International Journal of Computer Vision, 2014., Feb. 2015
  4. [4]
    Chai X, Li G, Lin Y, Xu Z, Tang Y, Chen X, Zhou M. Sign language recognition and translation with Kinect. In Proc. IEEE International Conference on Automatic Face and Gesture Recognition, April 2013.Google Scholar
  5. [5]
    Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2):91–110.CrossRefGoogle Scholar
  6. [6]
    Johnson A E, Hebert M. Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1999, 21(5):433–449.CrossRefGoogle Scholar
  7. [7]
    Morisset B, Rusu R B, Sundaresan A, Hauser K, Agrawal M, Latombe J C, Beetz M. Leaving flatland: Toward realtime 3D navigation. In Proc. IEEE International Conference on Robotics and Automation, May 2009, pp.3786–3793.Google Scholar
  8. [8]
    Hinterstoisser S, Holzer S, Cagniart C et al. Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes. In Proc. IEEE International Conference on Computer Vision (ICCV), Nov. 2011, pp.858–865.Google Scholar
  9. [9]
    Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. In Proc. Neural Information Processing Systems, Dec. 2012.Google Scholar
  10. [10]
    Zhang Z, Zhou C, Xin B,Wang Y, Gao W. An interactive system of stereoscopic video conversion. In Proc. the 20th ACM International Conference on Multimedia, Oct. 29–Nov. 2, 2012, pp.149–158.Google Scholar
  11. [11]
    Izadi S, Kim D, Hilliges O et al. KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera. In Proc. the 24th Annual ACM Symposium on User Interface Software and Technology, Nov. 2011, pp.559–568.Google Scholar
  12. [12]
    Liu S,Wang S,Wu L, Jiang S. Multiple feature fusion based hand-held object recognition with RGB-D data. In Proc. International Conference on Internet Multimedia Computing and Service, July 2014, p.303.Google Scholar
  13. [13]
    Lv X, Wang S, Li X, Jiang S. Combining heterogenous features for 3D handheld object recognition. In Proc. SPIE Optoelectronic Imaging and Multimedia Technology III, Oct. 2014.Google Scholar
  14. [14]
    Rivera-Rubio J, Idrees S, Alexiou I, Hadjilucas L, Bharath A. Small hand-held object recognition test (short). In Proc. the 2014 IEEE Winter Conference on Applications of Computer Vision (WACV), March 2014, pp.524–531.Google Scholar
  15. [15]
    Beck C, Broun A, Mirmehdi M, Pipe A, Melhuish C. Text line aggregation. In Proc. International Conference on Pattern Recognition Applications and Methods (ICPRAM), Mar. 2014, pp.393–401.Google Scholar
  16. [16]
    Silberman N, Hoiem D, Kohli P, Fergus R. Indoor segmentation and support inference from RGBD images. In Proc. the 12th ECCV, Part 5, Oct. 2012, pp.746-760Google Scholar
  17. [17]
    Koppula H S, Anand A, Joachims T, Saxena A. Semantic labeling of 3D point clouds for indoor scenes. In Proc. the 25th Neural Information Processing Systems, Dec. 2011.Google Scholar
  18. [18]
    Kanezaki A, Suzuki T, Harada T, Kuniyoshi Y. Fast object detection for robots in a cluttered indoor environment using integral 3D feature table. In Proc. the 2011 IEEE International Conference on Robotics and Automation (ICRA), May 2011, pp.4026–4033.Google Scholar
  19. [19]
    Felzenszwalb P F, Girshick R B, McAllester D, Ramanan D. Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(9):1627–1645.CrossRefGoogle Scholar
  20. [20]
    Alexandre L A. 3D object recognition using convolutional neural networks with transfer learning between input channels. In Proc. the 13th International Conference on Intelligent Autonomous Systems, July 2014.Google Scholar
  21. [21]
    Gupta S, Girshick R, Arbel´aez P, Malik J. Learning rich features from RGB-D images for object detection and segmentation. In Proc. the 13th ECCV, Part 7, Sept. 2014, pp.345–360.Google Scholar
  22. [22]
    Cimpoi M, Maji S, Kokkinos I, Mohamed S, Vedaldi A. Describing textures in the wild. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, June 2014, pp.3606-3613Google Scholar
  23. [23]
    Xiao J, Ehinger K, Hays J, Torralba A, Oliva A. SUN database: Exploring a large collection of scene categories. International Journal of Computer Vision, 2014., Feb. 2015.
  24. [24]
    Fu Y, Cao L, Guo G, Huang T S. Multiple feature fusion by subspace learning. In Proc. the 2008 International Conference on Content-Based Image and Video Retrieval, July 2008, pp.127–134.Google Scholar
  25. [25]
    Sun Q S, Jin Z, Heng P A, Xia D S. A novel feature fusion method based on partial least squares regression. In Proc. the 3rd International Conference on Advances in Pattern Recognition, Part 1, Aug. 2005, pp.268–277.Google Scholar
  26. [26]
    Barker M, Rayens W. Partial least squares for discrimination. Journal of Chemometrics, 2003, 17(3):166–173.CrossRefGoogle Scholar
  27. [27]
    Wohlkinger W, Vincze M. Ensemble of shape functions for 3D object classification. In Proc. the 2011 IEEE International Conference on Robotics and Biomimetics (ROBIO), Dec. 2011, pp.2987–2992.Google Scholar
  28. [28]
    Kanezaki A, Marton Z C, Pangercic D, Harada T, Kuniyoshi Y, Beetz M. Voxelized shape and color histograms for RGBD. In Proc. IROS Workshop on Active Semantic Perception and Object Search in the Real World, Sept. 2011.Google Scholar
  29. [29]
    Jia Y, Shelhamer Evan, Donahue J et al. Caffe: Convolutional architecture for fast feature embedding. arXiv:1408.5093, 2014., Feb. 2015.
  30. [30]
    Marton Z C, Pangercic D, Rusu R B, Holzbach A, Beetz M. Hierarchical object geometric categorization and appearance classification for mobile manipulation. In Proc. the 10th IEEE-RAS International Conference on Humanoid Robots, Dec. 2010, pp.365-370Google Scholar
  31. [31]
    Snoek C G, Worring M, Smeulders A W. Early versus late fusion in semantic video analysis. In Proc. the 13th Annual ACM International Conference on Multimedia, Nov. 2005, pp.399–402.Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Xiong Lv
    • 1
  • Shu-Qiang Jiang
    • 1
    Email author
  • Luis Herranz
    • 1
  • Shuang Wang
    • 1
  1. 1.Key Laboratory of Intelligent Information Processing of Chinese Academy of SciencesInstitute of Computing Technology Chinese Academy of SciencesBeijingChina

Personalised recommendations