Skip to main content
SpringerLink
Log in

Single and Multiple View Support Order Prediction in Clutter for Manipulation

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Robotic manipulation of objects in clutter remains a challenging problem to date. The challenge is posed by various levels of complexity involved in interaction among objects. Understanding these semantic interactions among different objects is important to manipulate in complex settings. It can play a significant role in extending the scope of manipulation to cluttered environment involving generic objects, and both direct and indirect physical contact. In our work, we aim at learning semantic interaction among objects of generic shapes and sizes lying in clutter involving physical contact. We infer three types of support relationships: “support from below”, “support from side”, and “containment”. Subsequently, the learned semantic interaction or support relationship is used to derive a sequence or order in which the objects surrounding the object of interest should be removed without causing damage to the environment. The generated sequence is called support order. We also extend understanding of semantic interaction from single view to multiple views and predict support order in multiple views. Using multiple views addresses those cases that are not handled when using single view such as scenarios of occlusion or missing support relationships. We have created two RGBD datasets for our experiments on support order prediction in single view and multiple views respectively. The datasets contains RGB images, point clouds and depth maps of various objects used in day-to-day life present in clutter with physical contact and overlap. We captured many different cluttered settings involving different kinds of object-object interaction and successfully learned support relationship and performed Support Order Prediction in these settings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Anand, A., Koppula, H.S., Joachims, T., Saxena, A.: Contextually guided semantic labeling and search for three-dimensional point clouds. Int. J. Robot. Res. (2012)

  2. Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(5), 898–916 (2011)

    Article  Google Scholar 

  3. Berenson, D., Diankov, R., Nishiwaki, K., Kagami, S., Kuffner, J.: Grasp planning in complex scenes. In: 2007 7th IEEE-RAS international conference on humanoid robots (2007)

  4. Berenson, D., Srinivasa, S.: Grasp synthesis in cluttered environments for dexterous hands. In: IEEE-RAS international conference on humanoid robots (Humanoids08) (2008)

  5. Bjorkman, M., Kragic, D.: Active 3D scene segmentation and detection of unknown objects. In: IEEE international conference on robotics and automation. IEEE (2010)

  6. Chang, L.Y., Srinivasa, S., Pollard, N.: Planning pre-grasp manipulation for transport tasks. In: IEEE international conference on robotics and automation (2010)

  7. Delong, A., Gorelick, L., Schmidt, F.R., Veksler, O., Boykov, Y.: Interactive segmentation with super-labels. In: International conference on energy minimization methods in computer vision and pattern recognition (2011)

  8. Dogar, M., Hsiao, K., Ciocarlie, M., Srinivasa, S.: Physics-based grasp planning through clutter. In: Robotics: science and systems VIII (2012)

  9. Dogar, M., Srinivasa, S.: A framework for push-grasping in clutter. In: Robotics: science and systems VII (2011)

  10. Fichtl, S., Alexander, J., Kraft, D., Jrgensen, J., Krger, N., Guerin, F.: Learning object relationships which determine the outcome of actions. Paladyn (2012)

  11. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM (1981)

  12. Harris, C., Stephens, M.: A combined corner and edge detector. In: Alvey vision conference, Manchester (1988)

  13. Hauser, K.: Cutting through the clutter: identifying minimally disturbed subsets. In: Robotics: science and systems workshop on robots in clutter: manipulation, perception and navigation in human environments (2012)

  14. Hoiem, D., Efros, A.A., Hebert, M.: Recovering occlusion boundaries from an image. Int. J. Comput. Vis. 91(3), 328–346 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  15. Katz, D., Kazemi, M., Bagnell, J.A., Stentz, A.: Clearing a pile of unknown objects using interactive perception. Tech. rep., DTIC Document (2012)

  16. Katz, D., Kazemi, M., Bagnell, J.A., Stentz, A.: Semi-autonomous manipulation of natural objects (2012)

  17. Koppula, H.S., Anand, A., Joachims, T., Saxena, A.: Semantic labeling of 3d point clouds for indoor scenes. In: Conference on neural information processing systems (2011)

  18. Kovesi, P.D.: MATLAB and Octave functions for computer vision and image processing. Available from: http://www.csse.uwa.edu.au/~pk/research/matlabfns/

  19. Lee, S., Kim, J., Lee, M., Yoo, K., Barajas, L.G., Menassa, R.: 3d visual perception system for bin picking in automotive sub-assembly automation. In: 2012 IEEE international conference on automation science and engineering (CASE) (2012)

  20. Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. ACM Trans. Graph. (TOG) 23(3), 689–694 (2004)

    Article  Google Scholar 

  21. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  22. Micelli, V., Strabala, K., Srinivasa, S.: Perception and control challenges for effective human-robot handoffs. In: Robotics: science and systems 2011 RGB-D workshop (2011)

  23. Mishra, A.K., Shrivastava, A., Aloimonos, Y.: Segmenting “simple” objects using RGB-D. IEEE Int. Conf. Robot. Autom. (2012)

  24. Oh, J.K., Lee, S., Lee, C.H.: Stereo vision based automation for a bin-picking solution. Int. J. Control. Autom. Syst. (2012)

  25. Panda, S., Abdul Hafez, A.H., Jawahar, C.V.: Learning support order for manipulation in clutter. In: IEEE/RSJ international conference on intelligent robots and systems (IROS) (2013)

  26. Ramisa, A., Aldavert, D., Vasudevan, S., Toledo, R., Lopez de Mantaras, R.: Evaluation of three vision based object perception methods for a mobile robot. J. Intell. Robot. Syst. (2011)

  27. Rasolzadeh, B., Björkman, M., Huebner, K., Kragic, D.: An active vision system for detecting, fixating and manipulating objects in the real world. Int. J. Robot. Res. 29(2–3), 133–154 (2010)

  28. Rosman, B., Ramamoorthy, S.: Learning spatial relationships between objects. Int. J. Robot. Res. 30(11), 1328–1342 (2011)

    Article  Google Scholar 

  29. Silberman, N., Fergus, R.: Indoor scene segmentation using a structured light sensor. In: Proceedings of the international conference on computer vision - workshop on 3D representation and recognition (2011)

  30. Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from RGBD images, pp. 746–760. Springer (2012)

  31. Panda, S., Abdul Hafez, A.H., Jawahar, C.V.: Learning semantic interaction among graspable objects. In: Pattern recognition and machine intelligence (2013)

  32. Sjoo, K., Jensfelt, P.: Learning spatial relations from functional simulation. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 1513–1519. IEEE (2011)

  33. Spenrath, F., Spiller, A., Verl, A.: Gripping point determination and collision prevention in a bin-picking application. In: 7th German Conference on Robotics, Proceedings of ROBOTIK 2012 (2012)

  34. Varadarajan, K., Potapova, E., Vincze, M.: Attention driven grasping for clearing a heap of objects (2012)

  35. Vedaldi, A., Fulkerson, B.: VLFeat: an open and portable library of computer vision algorithms (2008). http://www.vlfeat.org/

  36. Ye, J., Hua, K.A.: Exploiting depth camera for 3d spatial relationship interpretation. In: Proceedings of the 4th ACM multimedia systems conference, MMSys ’13 (2013)

  37. Russell, B.C., Torralba, A., Murphy, K.P., Freeman, W.T.: Labelme: a database and web-based tool for image annotation. IJCV (2008)

  38. Jia, Z., Gallagher, A., Saxena, A., Chen, T.: 3D-based reasoning with blocks, support, and stability. CVPR (2013)

  39. Chang, L., Smith, J., Fox, D.: Interactive singulation of objects from a pile. ICRA (2012)

  40. Katz, D., Kazemi, M., Bagnell, J., Stentz, A.: Clearing a pile of unknown objects using interactive perception. ICRA (2013)

  41. Mojtahedzadeh, R., Bouguerra, A., Lilienthal, A.: Automatic relational scene representation for safe robotic manipulation tasks. IROS (2013)

  42. Ornan, O., Degani, A.: Toward autonomous disassembling of randomly piled objects with minimal perturbation. IROS (2013)

  43. Aleotti, J., Caselli, S.: Efficient planning of disassembly sequences in physics-based animation. IROS (2009)

  44. Stilman, M., Schamburek, J.-U., Kuffner, J., Asfour, T.: Manipulation planning among movable obstacles. ICRA (2007)

Download references

Author information

Authors and Affiliations

  1. International Institute of Information Technology, Hyderabad, India

    Swagatika Panda & C. V. Jawahar

  2. Department of Computer Engineering, Hasan Kalyoncu University, Sahinbey, Gaziantep, Turkey

    A. H. Abdul Hafez

Authors
  1. Swagatika Panda
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. H. Abdul Hafez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. C. V. Jawahar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Swagatika Panda.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Panda, S., Abdul Hafez, A.H. & Jawahar, C.V. Single and Multiple View Support Order Prediction in Clutter for Manipulation. J Intell Robot Syst 83, 179–203 (2016). https://doi.org/10.1007/s10846-015-0330-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-015-0330-z

Keywords

Search

Navigation