Journal of Real-Time Image Processing

, Volume 10, Issue 4, pp 667–682 | Cite as

Textured/textureless object recognition and pose estimation using RGB-D image

  • Wei WangEmail author
  • Lili Chen
  • Ziyuan Liu
  • Kolja Kühnlenz
  • Darius Burschka
Special Issue Paper


In this paper, we propose a novel global object descriptor, so-called Viewpoint oriented Color-Shape Histogram (VCSH), which combines 3D object’s color and shape features. The descriptor is efficiently used in a real-time textured/textureless object recognition and 6D pose estimation system, while also applied for object localization in a coherent semantic map. We build the object model first by registering from multi-view color point clouds, and generate partial-view object color point clouds from different synthetic viewpoints. Thereafter, the extracted color and shape features are correlated as a VCSH to represent the corresponding object patch data. For object recognition, the object can be identified and its initial pose is estimated through matching within our built database. Afterwards the object pose can be optimized by utilizing an iterative closest point strategy. Therefore, all the objects in the observed area are finally recognized and their corresponding accurate poses are retrieved. We validate our approach through a large number of experiments, including daily complex scenarios and indoor semantic mapping. Our method is proven to be efficient by guaranteeing high object recognition rate, accurate pose estimation result as well as exhibiting the capability of dealing with environmental illumination changes.


Real-time robotic vision Object recognition and pose estimation Viewpoint oriented color–shape histogram Semantic map 

Supplementary material

MOV (21022 KB)


  1. 1.
    Biederman, I., Gerhardstein, P.C.: Recognizing depth-rotated objects: Evidence and conditions for three-dimensional viewpoint invariance. J. Exp. Psychol. Hum. Percept. Perform. 19(6), 1162–1182 (1993)CrossRefGoogle Scholar
  2. 2.
    Biederman, I., Cooper, E.E.: Evidence for complete translational and reflectional invariance in visual object priming. Perception 20(5), 585–593 (1991)CrossRefGoogle Scholar
  3. 3.
    Edelman, S., Bülthoff, H.H.: Orientation dependence in the recognition of familiar and novel view of three-dimensional objects. Vis. Res. 32(12), 2385–2400 (1992)CrossRefGoogle Scholar
  4. 4.
    Ellis, R., Allport, D.A., Humphreys, G.W., Collis, J.: Varieties of object constancy. Q. J. Exp. Psychol. 41(4), 775–796 (1989)CrossRefGoogle Scholar
  5. 5.
    Fiser, J., Biederman, I.: Size invariance in visual object priming of gray-scale images. Perception 24(7), 741–748 (1995)CrossRefGoogle Scholar
  6. 6.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. IJCV 60(2), 91–110 (2004)CrossRefGoogle Scholar
  7. 7.
    Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: Speeded-up robust features. CVIU 110(3), 346–359 (2004)Google Scholar
  8. 8.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of CVPR, pp. 886–893 (2005)Google Scholar
  9. 9.
    Abdel-Hakim, A.E., Farag, A.A.: CSIFT: A SIFT descriptor with color invariant characteristics. In: Proceedings of CVPR, pp. 1978–1983 (2006)Google Scholar
  10. 10.
    Gevers, T., Smeulders, A.W.M.: Color-based object recognition. IJPR 32, 453–464 (1999)Google Scholar
  11. 11.
    Gevers, T.: Robust histogram construction from color invariants for object recognition. TPAMI 26(1), 113–118 (2004)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Crandall, D., Luo, J.: Robust color object detection using spatial-color joint probability functions. In: Proceedings of CVPR, pp. 1443–1453 (2004)Google Scholar
  13. 13.
    Mitra, N.J., Guibas, L.J., Giesen, J., Pauly, M.: Probabilistic fingerprints for shapes. In: Proceedings of Eurographics Symposium on Geometry Processing, pp. 121–130 (2006)Google Scholar
  14. 14.
    Clarenz, U., Rumpf, M., Telea, A.: Robust feature detection and local classification for surfaces based on moment analysis. IEEE Trans. Vis. Comput. Graph. 10(5), 516–524 (2004)CrossRefGoogle Scholar
  15. 15.
    Gelfand, N., Mitra, N.J., Guibas, L.J., Pottmann, H.: Robust global registration. In: Proceedings of Eurographics Symposium on Geometry Processing, no. 197 (2005)Google Scholar
  16. 16.
    Safronov, K., Tchouchenkov, I., Wrn, H.: Hierarchical iterative pattern recognition method for solving bin picking problem. In: Proceedings of Robotik, pp. 3–6 (2008)Google Scholar
  17. 17.
    Shilane, P., Funkhauser, T.: Selecting distinctive 3D shape descriptors for similarity retrieval. In: Proceedings of IEEE Conference on Shape Modeling and Applications, pp. 18–27 (2006)Google Scholar
  18. 18.
    Pauly, M., Keiser, R., Gross, M.: Multi-scale feature extraction on point-sampled surfaces. Comput. Graph. Forum 22(3), 281–289 (2003)CrossRefGoogle Scholar
  19. 19.
    Kolomenkin, M., Shimshoni, I., Tal, A.: On edge detection on surfaces. In: Proceedings of CVPR, pp. 2767–2774 (2009)Google Scholar
  20. 20.
    Tang, J., Miller, S., Singh, A., Abbeel, P.: A textured object recognition pipeline for color and depth image data. In: Proceedings of ICRA, pp. 3467–3474 (2012)Google Scholar
  21. 21.
    Kanezaki, A., Marton, Z., Pangercic, D., Harada, T., Kuniyoshi, Y., Beetz, M.: Voxelized shape and color histograms for RGB-D. In: Proceedings of IROS, Workshop on Active Semantic Perception and Object Search in the Real World (2011)Google Scholar
  22. 22.
    Kanezaki, A., Harada, T., Kuniyoshi, Y.: Partial matching of real textured 3D objects using color cubic higher-order local auto-correlation features. J. Vis. Comput. 26(10), 1269–1281 (2010)CrossRefGoogle Scholar
  23. 23.
    Choi, C., Christensen, H.I.: 3D pose estimation of daily objects using an RGB-D camera. In: Proceedings of IROS, pp. 3342–3349 (2012)Google Scholar
  24. 24.
    Tombari, F., Salti, S., Stefano, L.D.: A combined texture–shape descriptor for enhanced 3D feature matching. In: Proceedings of ICIP, pp. 809–812 (2011)Google Scholar
  25. 25.
    Liebelt, J., Schmid, C., Schertler, K. Viewpoint-independent object class detection using 3D feature maps. In: Proceedings of CVPR, pp. 1–8 (2008)Google Scholar
  26. 26.
    Fan, Z., Lu, B.: Fast recognition of multi-view faces with feature selection. In: Proceedings of ICCV, pp. 76–81 (2005)Google Scholar
  27. 27.
    Kushal, A., Schmid, C., Ponce, J.: Flexible object models for category-level 3D object recognition. In: Proceedings of CVPR, pp. 1–8 (2007)Google Scholar
  28. 28.
    Thomas, A., Ferrari, V., Leibe, B., Tuytelaars, T., Schiele, B., Gool, L.V.: Towards multi-view object class detection. In: Proceedings of CVPR, pp. 1589–1596 (2006)Google Scholar
  29. 29.
    Rusu, R.B., Bradski, G., Thibaux, R., Hsu, J.: Fast 3D recognition and pose using the viewpoint feature histogram. In: Proceedings of IROS, pp. 3467–3474 (2010)Google Scholar
  30. 30.
    Hinterstoisser, S., Lepetit, V., Ilic, S., Holzer, S., Bradski, G., Konolige, K., Navab, N.: Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes. In: Proceedings of ACCV, pp. 548–562 (2012)Google Scholar
  31. 31.
    Rusu, R.B., Blodow, N., Beetz, M.: Fast point feature histograms (FPFH) for 3D registration. In: Proceedings of ICRA, pp. 3212–3217 (2009)Google Scholar
  32. 32.
    Wohlkinger, W., Vincze, M.: Ensemble of shape functions for 3D object classification. In: Proceedings of ROBIO, pp. 2987–2992 (2011)0Google Scholar
  33. 33.
    Tombari, F., Salti, S., Stefano, L.D.: Unique signatures of histograms for local surface description. In: Proceedings of ECCV, pp. 356–369 (2010)Google Scholar
  34. 34.
    Vadivel, A., Majumdar, A.K., Sural, S.: Perceptually smooth histogram generation from the HSV color space for content based image retrieval. In: Proceedings of Advances in Pattern Recognition, pp. 248–251 (2003)Google Scholar
  35. 35.
    Akgül, C.B., Sankur, B., Schmitt, F., Yemez, Y.: Multivariate density-based 3D shape descriptors. In: Proceedings of Shape Modeling International, pp. 3–12 (2007)Google Scholar
  36. 36.
    Alexa, M., Behr, J., Cohen-Or, D., Fleishman, S., Levin, D., Silva, C.T.: Computing and rendering point set surfaces. IEEE Trans. Vis. Comput. Graph. 9(1), 3–15 (2003)CrossRefGoogle Scholar
  37. 37.
    Zhang, Z.: Iterative point matching for registration of free-form curves and surfaces. IJCV 13(2), 119–152 (1994)CrossRefGoogle Scholar
  38. 38.
    Arun, K.S., Huang, T.S., Blostein, S.D.: Least-squares fitting of two 3-D point sets. TPAMI 9(5), 698–700 (1987)CrossRefGoogle Scholar
  39. 39.
    Sural, S., Qian, G., Pramanik, S.: A histogram with perceptually smooth color transition for image retrieval. In: Proceedings of International Conference on Computer Vision, Pattern Recognition and Image Processing, pp. 664–667 (2002)Google Scholar
  40. 40.
    Eberhardt, H., Klumpp, V., Hanebeck, U.D.: Density trees for efficient nonlinear state estimation. In: Proceedings of International Conference on Information Fusion, pp. 1–8 (2010)Google Scholar
  41. 41.
    Wang, W., Koropouli, V., Dongheui, L., Khnlenz, K.: Articulated object modeling based on visual and haptic observations. In: Proceedings of VISAPP, pp. 253–259 (2013)Google Scholar
  42. 42.
    Wang, W., Brščić, D., He, Z., Hirche, S., Kühnlenz, K.: Real-time human body motion estimation based on multi-layer laser scans. In: Proceedings of URAI, pp. 297–302 (2011)Google Scholar
  43. 43.
    Wang, W., Li, S., Chen, L., Chen, D., Kühnlenz, K.: Fast object recognition and 6D pose estimation using viewpoint oriented color–shape histogram. In: Proceedings of ICME, pp. 1–6 (2013)Google Scholar
  44. 44.
    Grisetti, G., Stachniss, C., Burgard, W.: Improved techniques for grid mapping with Rao–Blackwellized particle filters. IEEE Trans. Rob. 23(1), 34–46 (2007)CrossRefGoogle Scholar
  45. 45.
    Liu, Z., Wang, W., Chen, D., Wichert, G.v.: A coherent semantic mapping system based on parametric environment abstraction and 3D object localization. In: Proceedings of European Conference on Mobile Robots (2013)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Wei Wang
    • 1
    Email author
  • Lili Chen
    • 1
  • Ziyuan Liu
    • 2
  • Kolja Kühnlenz
    • 3
  • Darius Burschka
    • 1
  1. 1.Institute of Robotics and Embedded SystemsTechnische Universität MünchenGarching bei MünchenGermany
  2. 2.Institute of Automatic Control EngineeringTechnische Universität MünchenMunichGermany
  3. 3.Institute of Advanced StudyTechnische Universität MünchenMunichGermany

Personalised recommendations