Journal of Intelligent & Robotic Systems

, Volume 76, Issue 1, pp 35–56 | Cite as

Part-Based Geometric Categorization and Object Reconstruction in Cluttered Table-Top Scenes

Paper Type: Categories (7) and (5)
  • Zoltan-Csaba Marton
  • Ferenc Balint-Benczedi
  • Oscar Martinez Mozos
  • Nico Blodow
  • Asako Kanezaki
  • Lucian Cosmin Goron
  • Dejan Pangercic
  • Michael Beetz


This paper presents an approach for 3D geometry-based object categorization in cluttered table-top scenes. In our method, objects are decomposed into different geometric parts whose spatial arrangement is represented by a graph. The matching and searching of graphs representing the objects is sped up by using a hash table which contains possible spatial configurations of the different parts that constitute the objects. Additive feature descriptors are used to label partially or completely visible object parts. In this work we categorize objects into five geometric shapes: sphere, box, flat, cylindrical, and disk/plate, as these shapes represent the majority of objects found on tables in typical households. Moreover, we reconstruct complete 3D models that include the invisible back-sides of objects as well, in order to facilitate manipulation by domestic service robots. Finally, we present an extensive set of experiments on point clouds of objects using an RGBD camera, and our results highlight the improvements over previous methods.


Object categorization 3D geometry Part-graph hashing Clutter Domestic robotics 

Mathematics Subject Classifications (2010)

68 60 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aldoma, A., Blodow, N., Gossow, D., Gedikli, S., Rusu, R., Vincze, M., Bradski, G.: CAD-model recognition and 6 DOF pose estimation using 3D cues. In: ICCV Workshop on 3D Representation and Recognition (3dRR11). Barcelona (2011)Google Scholar
  2. 2.
    Aldoma, A., Marton, Z.C., Tombari, F., Wohlkinger, W., Potthast, C., Zeisl, B., Rusu, R.B., Gedikli, S., Vincze, M.: Tutorial: point cloud library–three-dimensional object recognition and 6 DoF pose estimation. Robot. Autom. Mag. 19(3), 80–91 (2012)CrossRefGoogle Scholar
  3. 3.
    Balint-Benczedi, F.,Marton, Z.C., Beetz, M.: Efficient part-graph hashes for object categorization. In: 5th International Conference on Cognitive Systems (CogSys) (2012)Google Scholar
  4. 4.
    Bergström, N., Björkman, M., Kragic, D.: Generating object hypotheses in natural scenes through human-robot interaction. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 827–833 (2011)Google Scholar
  5. 5.
    Bersch, C., Pangercic, D., Osentoski, S., Hausman, K., Marton, Z.C., Ueda, R., Okada, K., Beetz, M.: Segmentation of textured and textureless objects through interactive perception. In: RSS Workshop on Robots in Clutter: Manipulation, Perception and Navigation in Human Environments. Sydney (2012)Google Scholar
  6. 6.
    Biederman, I.: Recognition-by-components. A theory of human image understanding. Psychol. Rev. (1987)Google Scholar
  7. 7.
    Blodow, N., Goron, L.C., Marton, Z.C., Pangercic, D., Rühr, T., Tenorth, M., Beetz, M.: Autonomous semantic mapping for robots performing everyday manipulation tasks in kitchen environments. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). San Francisco (2011)Google Scholar
  8. 8.
    Comaniciu, D., Meer, P., Member, S.: Mean shift: a robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24, 603–619 (2002)CrossRefGoogle Scholar
  9. 9.
    Dickinson, S.: The evolution of object categorization and the challenge of image abstraction. In: Dickinson, S. Leonardis, A., Schiele, B., Tarr, M. (eds) Object Categorization: Computer and Human Vision Perspectives (2009)Google Scholar
  10. 10.
    Felzenszwalb, P.F., Girshick, R.B., McAllester, D.A., Ramanan, D.: Object detection with discriminatively trained part-based models. IEEE Trans. Pattern Anal. Mach. Intell. 32(9) (2010)Google Scholar
  11. 11.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation. Int. J. Comput. Vis. 59(2), 167–181. doi: 10.1023/B:VISI.000002228819776.77
  12. 12.
    Fergus, R., Perona, P., Zisserman, A.: Object class recognition by unsupervised scale-invariant learning. In: In CVPR, pp. 264–271 (2003)Google Scholar
  13. 13.
    Fowlkes, C.C., Martin, D.R., Malik, J.: Local figure-ground cues are valid for natural images. J. Vis. 7(8) (2007)Google Scholar
  14. 14.
    Gomes, R., Krause, A., Perona, P.: Discriminative clustering by regularized information maximization. Adv. Neural Inf. Process. Syst. 23, 1–9 (2010)Google Scholar
  15. 15.
    Goron, L.C., Marton, Z.C., Lazea, G., Beetz, M.: Segmenting cylindrical and box-like objects in cluttered 3D scenes. In: 7th German Conference on Robotics (ROBOTIK 2012). Munich (2012)Google Scholar
  16. 16.
    Hausman, K., Balint-Benczedi, F., Pangercic, D., Marton, Z.C., Ueda, R., Okada, K., Beetz, M.: Tracking-based interactive segmentation of textureless objects. In: IEEE International Conference on Robotics and Automation (ICRA). Best Service Robotics Paper Award Finalist. Karlsruhe (2013)Google Scholar
  17. 17.
    Huber, D., Kapuria, A., Donamukkala, R.R., Hebert, M.: Parts-based 3d object classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 04) (2004)Google Scholar
  18. 18.
    Jacobs, D.W.: Perceptual organization as generic object recognition. In: From Fragments to Objects - Segmentation and Grouping in Vision, chap. IV. Models Of Segmentation And Grouping, pp. 295–329 (2001)Google Scholar
  19. 19.
    Kanezaki, A., Marton, Z.C., Pangercic, D., Harada, T., Kuniyoshi, Y., Beetz, M.: Voxelized shape and color histograms for RGB-D. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Workshop on Active Semantic Perception and Object Search in the Real World. San Francisco (2011)Google Scholar
  20. 20.
    Kanezaki, A., Nakayama, H., Harada, T., Kuniyoshi, Y.: High-speed 3d object recognition using additive features in a linear subspace. In: Proceedings of International Conference on Robotics and Automation (ICRA), pp. 3128–3134 (2010)Google Scholar
  21. 21.
    Kanezaki, A., Suzuki, T., Harada, T., Kuniyoshi, Y.: Fast object detection for robots in a cluttered indoor environment using integral 3D feature table. In: Proceedings of the IEEE ICRA (2011)Google Scholar
  22. 22.
    Lai, K., Bo, L., Ren, X., Fox, D.: A large-scale hierarchical multi-view rgb-d object dataset. In: Proceedings of International Conference on Robotics and Automation (ICRA) (2011)Google Scholar
  23. 23.
    Lai, K., Bo, L., Ren, X., Fox, D.: Sparse distance learning for object recognition combining rgb and depth information. In: Proceedings of International Conference on Robotics and Automation (ICRA) (2011)Google Scholar
  24. 24.
    Lai, K., Fox, D.: Object recognition in 3d point clouds using web data and domain adaptation. Int. J. Robot. Res. 29(8), 1019–1037 (2010). doi: 10.1177/0278364910369190 CrossRefGoogle Scholar
  25. 25.
    Lam, L., Suen, C.Y.: Optimal combinations of pattern classifiers. Pattern Recognit. Lett 16(9), 945–954 (1995)CrossRefGoogle Scholar
  26. 26.
    Leibe, B., Leonardis, A., Schiele, B.: Robust object detection with interleaved categorization and segmentation. Int. J. Comput. Vis. 77(1–3), 259–289 (2008)CrossRefGoogle Scholar
  27. 27.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 91–110 (2004)CrossRefGoogle Scholar
  28. 28.
    Malisiewicz, T., Efros, A.A.: Improving spatial support for objects via multiple segmentations. In: Proceedings of the British Machine Vision Conference (2007)Google Scholar
  29. 29.
    Marton, Z.C., Balint-Benczedi, F., Blodow, N., Goron, L.C., Beetz, M.: Object categorization in clutter using additive features and hashing of part-graph descriptors. In: Proceedings of Spatial Cognition 2012. Abbey Kloster Seeon (2012)Google Scholar
  30. 30.
    Marton, Z.C., Balint-Benczedi, F., Mozos, O.M., Pangercic, D., Beetz, M.: Cumulative object categorization in clutter. In: 2nd Workshop on Robotics in Clutter, at Robotics: Science and Systems (RSS) (2013)Google Scholar
  31. 31.
    Marton, Z.C., Pangercic, D., Blodow, N., Beetz, M.: Combined 2D-3D categorization and classification for multimodal perception systems. Int. J. Robot. Res. (2011)Google Scholar
  32. 32.
    Marton, Z.C., Pangercic, D., Rusu, R.B., Holzbach, A., Beetz, M.: Hierarchical object geometric categorization and appearance classification for mobile manipulation. In: Proceedings of 2010, IEEE-RAS International Conference on Humanoid Robots. Nashville (2010)Google Scholar
  33. 33.
    Marton, Z.C., Rusu, R.B., Jain, D., Klank, U., Beetz, M.: Probabilistic categorization of kitchen objects in table settings with a composite sensor. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. USA (2009)Google Scholar
  34. 34.
    Marton, Z.C., Seidel, F., Balint-Benczedi, F., Beetz, M.: Ensembles of strong learners for multi-cue classification. Pattern Recognit. Lett. Special Issue on Scene Understandings and Behaviours Analysis (2012)Google Scholar
  35. 35.
    Mishra, A.K., Aloimonos, Y.: Visual segmentation of “Simple” objects for robots. In: Robotics: Science and Systems (RSS) (2011)Google Scholar
  36. 36.
    Mozos, O.M., Marton, Z.C., Beetz, M.: Furniture models learned from the www—using web catalogs to locate and categorize unknown furniture pieces in 3D laser scans. Robot. Autom. Mag. 18(2), 22–32 (2011)CrossRefGoogle Scholar
  37. 37.
    Mueller, C.A., Pathak, K., Birk, A.: Object recognition in rgbd images of cluttered environments using graph-based categorization with unsupervised learning of shape parts. In: International Conference on Intelligent Robots and Systems (IROS) (2013)Google Scholar
  38. 38.
    Pangercic, D., Haltakov, V., Beetz, M.: Fast and robust object detection in household environments using vocabulary trees with sift descriptors. Workshop on Active Semantic Perception and Object Search in the Real World. San Francisco (2011)Google Scholar
  39. 39.
    Richtsfeld, A., Morwald, T., Prankl, J., Zillich, M., Vincze, M.: Segmentation of unknown objects in indoor environments. In: Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference, pp. 4791–4796. doi: 10.1109/IROS.2012.6385661 (2012)
  40. 40.
    Ruiz-Correa, S., Shapiro, L.G., Meila, M.: A new paradigm for recognizing 3-D object shapes from range data. In: International Conference on Computer Vision (2003)Google Scholar
  41. 41.
    Rusu, R.B., Blodow, N., Marton, Z.C., Beetz, M.: Close-range scene segmentation and reconstruction of 3D point cloud maps for mobile manipulation in human environments. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). St. Louis (2009)Google Scholar
  42. 42.
    Rusu, R.B., Cousins, S.: 3D is here: point cloud library (PCL). In: IEEE International Conference on Robotics and Automation (ICRA). Shanghai (2011)Google Scholar
  43. 43.
    Rusu, R.B., Marton, Z.C., Blodow, N., Dolha, M., Beetz, M.: Towards 3D Point Cloud Based Object Maps for Household Environments. Robot. Auton. Syst. J. (Special Issue on Semantic Knowledge in Robotics) 56(11), 927–941 (2008)CrossRefGoogle Scholar
  44. 44.
    Scavino, E., Wahab, D.A., Basri, H., Mustafa, M.M., Hussain, A.: A Genetic Algorithm for the Segmentation of Known Touching Objects 5, 711–716 (2009)Google Scholar
  45. 45.
    Schnabel, R., Wahl, R., Klein, R.: Efficient ransac for point-cloud shape detection. In: Computer Graphics Forum, vol. 26, pp. 214–226. Wiley Online Library (2007)Google Scholar
  46. 46.
    Schuster, M., Okerman, J., Nguyen, H., Rehg, J., Kemp, C.: Perceiving clutter and surfaces for object placement in indoor environments. In: Proceedings of the 10th IEEE-RAS International Conference on Humanoid Robots (Humanoids), pp. 152–159 (2010) doi: 10.1109/ICHR.2010.5686328
  47. 47.
    Shotton, J., Blake, A., Cipolla, R.: Multiscale categorical object recognition using contour fragments. IEEE Trans. Pattern Anal. Mach. Intell. 30(7) (2008)Google Scholar
  48. 48.
    Shotton, J., Winn, J., Rother, C., Criminisi, A.: Textonboost for image understanding: multi-class object recognition and segmentation by jointly modeling texture, layout, and context. Int. J. Comput. Vis. (2007)Google Scholar
  49. 49.
    Singh, M., Hoffman, D.D.: Part-Based representations of visual shape and implications for visual cognition. In: From Fragments to Objects–Segmentation and Grouping in Vision, chap. IV. Models Of Segmentation And Grouping, pp. 401–459 (2001)Google Scholar
  50. 50.
    Socher, R., Lin, C.C.Y., Ng, A.Y., Manning, C.D.: Parsing natural scenes and natural language with recursive neural networks. In: 28th International Conference on Machine Learning, pp. 129–136 (2011)Google Scholar
  51. 51.
    Somanath, G., Rohith, M., Metaxas, D., Kambhamettu, C.: D-clutter: building object model library from unsupervised segmentation of cluttered scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2783–2789 (2009) doi: 10.1109/CVPR.2009.5206579
  52. 52.
    Sun, M., Bradski, G., Xu, B.X., Savarese, S.: Depth-encoded hough voting for joint object detection and shape recovery. In: Proceedings of the 11th European conference on Computer vision: part V, ECCV’10 pp. 658–671. Springer, Berlin, Heidelberg (2010).
  53. 53.
    Taylor, G., Kleeman, L.: Chapter 4: 3D object modelling and classification. In: Visual Perception and Robotic Manipulation–3D Object Recognition, Tracking and Hand-Eye Coordination, Springer Tracts in Advanced Robotics, vol. 26, pp.57–83. Springer, Berlin, Heidelberg (2006)Google Scholar
  54. 54.
    Tombari, F., Di Stefano, L.: Object recognition in 3d scenes with occlusions and clutter by hough voting. In: Proceedings of the Fourth Pacific-Rim Symposium on Image and Video Technology (PSIVT), pp. 349–355 (2010) doi: 10.1109/PSIVT.2010.65
  55. 55.
    Torres, M.M., Romea, A.C., Srinivasa, S.: MOPED: a scalable and low latency object recognition and pose estimation system. In: Proceedings of International Conference on Robotics and Automation (ICRA) (2010)Google Scholar
  56. 56.
    Triebel, R., Shin, J., Siegwart, R.: Segmentation and unsupervised part-based discovery of repetitive objects. In: Proceedings of Robotics: Science and Systems. Zaragoza (2010)Google Scholar
  57. 57.
    Vernon, D.: Cognitive vision: the case for embodied perception. In: Image and Vision Computing. Elsevier (2005)Google Scholar
  58. 58.
    Watanabe, S., Pakvasa, N.: Subspace method in pattern recognition. In: Proceedings of 1st International Joint Conference on Pattern Recognition (1973)Google Scholar
  59. 59.
    Yildirim, I., Jacobs, R.A.: Transfer of object category knowledge across visual and haptic modalities: experimental and computational studies. Cognition 126(2), 135–148 (2013). doi: 10.1016/j.cognition.2012.08.005 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2014

Authors and Affiliations

  • Zoltan-Csaba Marton
    • 1
  • Ferenc Balint-Benczedi
    • 2
  • Oscar Martinez Mozos
    • 3
  • Nico Blodow
    • 4
  • Asako Kanezaki
    • 5
  • Lucian Cosmin Goron
    • 4
  • Dejan Pangercic
    • 6
  • Michael Beetz
    • 2
  1. 1.Institute of Robotics and MechatronicsGerman Aerospace Center (DLR)OberpfaffenhofenGermany
  2. 2.Institute of Artificial IntelligenceUniversität Bremen, Center for Computing Technologies (TZI)BremenGermany
  3. 3.School of Computer ScienceUniversity of LincolnLincolnUK
  4. 4.Intelligent Autonomous SystemsTechnische Universität MünchenMünchenGermany
  5. 5.Machine Intelligence Lab, Deptartment of Mechano-Informatics, Graduate School of Information Science & TechnologyThe University of TokyoTokyoJapan
  6. 6.Autonomous Technologies Group Robert Bosch LLCPalo AltoUSA

Personalised recommendations