Learning Based Industrial Bin-Picking Trained with Approximate Physics Simulator

  • Ryo Matsumura
  • Kensuke HaradaEmail author
  • Yukiyasu Domae
  • Weiwei Wan
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 867)


In this research, we tackle the problem of picking an object from randomly stacked pile. Since complex physical phenomena of contact among objects and fingers makes it difficult to perform the bin-picking with high success rate, we consider introducing a learning based approach. For the purpose of collecting enough number of training data within a reasonable period of time, we introduce a physics simulator where approximation is used for collision checking. In this paper, we first formulate the learning based robotic bin-picking by using CNN (Convolutional Neural Network). We also obtain the optimum grasping posture of parallel jaw gripper by using CNN. Finally, we show that the effect of approximation introduced in collision checking is relaxed if we use exact 3D model to generate the depth image of the pile as an input to CNN.



This research was supported by NEDO (New Energy and Industrial Technology Development Organization).


  1. 1.
    Turkey, M.J.: Automated online measurement of limestone particle size distributions using 3D range data. J. Process Control 21, 254–262 (2011)CrossRefGoogle Scholar
  2. 2.
    Kristensen, S., et al.: Bin-picking with a solid state range camera. Robot. Auton. Syst. 35, 143–151 (2001)CrossRefGoogle Scholar
  3. 3.
    Ghita, O., Whelan, P.F.: A bin picking system based on depth from defocus. J. Mach. Vis. Appl. 13(4), 234–244 (2003)CrossRefGoogle Scholar
  4. 4.
    Kirkegaard, J., Moeslund, T.B.: Bin-picking based on harmonic shape contexts and graph-based matching. In: International Conference on Pattern Recognition, vol. 2, pp. 581–584 (2006)Google Scholar
  5. 5.
    Fuchs, S., et al.: Cooperative bin-picking with time-of-flight camera and impedance controlled DLR lightweight robot III. In: IEEE International Conference on Intelligent Robots and Systems, pp. 4862–4867 (2010)Google Scholar
  6. 6.
    Zuo, A., et al.: A hybrid stereo feature matching algorithm for stereo vision-based bin picking. J. Pattern Recognit. Artif. Intell. 18(8), 1407–1422 (2004)CrossRefGoogle Scholar
  7. 7.
    Domae, Y., et al.: Fast graspability evaluation on single depth maps for bin picking with general grippers. In: IEEE International Conference on Robotics and Automation, pp. 1197–2004 (2014)Google Scholar
  8. 8.
    Dupuis, D.C., et al.: Two-fingered grasp planning for randomized bin-picking. In: Robotics, Science and Systems 2008 Manipulation Workshop (2008)Google Scholar
  9. 9.
    Harada, K., et al.: Probabilistic approach for object bin picking approximated by cylinders. In: IEEE International Conference on Robotics and Automation, pp. 3727–3732 (2013)Google Scholar
  10. 10.
    Harada, K., et al.: Project on development of a robot system for random picking–grasp/manipulation planner for a dual-arm manipulator. In: IEEE/SICE International Symposium on System Integration, pp. 583–589 (2014)Google Scholar
  11. 11.
    Levine, S., et al.: Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. In: International Symposium on Experimental Robotics (2016)Google Scholar
  12. 12.
    Harada, K., et al.: Initial experiments on learning-based randomized bin-picking allowing finger contact with neighboring objects. In: IEEE International Conference on Automation Science and Engineering, pp. 1196–1202 (2016)Google Scholar
  13. 13.
    Zeng, A., et al.: Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross domain image matching.
  14. 14.
    Lin, G., et al.: RefineNet: multi-path refinement networks for high-resolution semantic segmentation.
  15. 15.
    Mahler, J., et al.: Dex-Net 2.0: deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics.
  16. 16.
    Bousmalis, K., et al.: Using simulation and domain adaptation to improve efficiency of deep robotic grasping (2017).
  17. 17.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)CrossRefGoogle Scholar
  18. 18.
    Curtis, N., Xiao, J.: Efficient and effective grasping of novel objects through learning and adapting a knowledge base. In: IEEE International Conference on Intelligent Robots and Systems, pp. 2252–2257 (2008)Google Scholar
  19. 19.
    Lenz, I., Lee, H., Saxena, A.: Deep learning for detecting robotic grasps. Int. J. Robot. Res. 34(4–5), 705–724 (2015)CrossRefGoogle Scholar
  20. 20.
    Pas, A.t., Platt, R.: Using geometry to detect grasps in 3D point clouds. In: International Symposium on Robotics Research (2015)Google Scholar
  21. 21.
    Ekvall, S., Kragic, D.: Learning and evaluation of the approach vector for automatic grasp generation and planning. In: IEEE International Conference on Robotics and Automation (2007)Google Scholar
  22. 22.
    Harada, K., Kaneko, K., Kanehiro, F.: Fast grasp planning for hand/arm systems based on convex model. In: IEEE International Conference on Robotics and Automation, pp. 1162–1168 (2008)Google Scholar
  23. 23.
    Harada, K., et al.: Stability of soft-finger grasp under gravity. In: IEEE International Conference on Robotics and Automation, pp. 883–888 (2014)Google Scholar
  24. 24.
    Thrun, S., et al.: Probabilistic Robotics. MIT Press, Cambridge (2005)zbMATHGoogle Scholar
  25. 25.
    Nagata, K., et al.: Picking up and indicated object in a complex environment. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (2010)Google Scholar
  26. 26.
    Aldoma, A., et al.: OUR-CVFH - oriented, unique and repeatable clustered viewpoint feature histogram for object recognition and 6DOF pose estimation. In: Pattern Recognition, pp. 113–122. Springer (2012)Google Scholar
  27. 27.
    Stein, C.M., et al.: Object partitioning using local convexity. In: IEEE International Conference on Computer Vision and Pattern Recognition (2014)Google Scholar
  28. 28.
    PCL-Point Cloud Library.
  29. 29.
    Mamou, K., Faouzi, G.: A simple and efficient approach for 3D mesh approximate convex decomposition. In: IEEE International Conference on Image Processing, pp. 3501–3504 (2009)Google Scholar
  30. 30.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)CrossRefGoogle Scholar
  31. 31.
    Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  32. 32.
    Grauman, K., Leibe, B.: Visual Object Recognition. Synthesis Lectures on Artificial Intelligence and Machine Learning, vol. 5, no. 2, pp. 1–181 (2011)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Ryo Matsumura
    • 1
  • Kensuke Harada
    • 1
    • 2
    Email author
  • Yukiyasu Domae
    • 2
  • Weiwei Wan
    • 1
    • 2
  1. 1.Graduate School of Engineering ScienceOsaka UniversitySuitaJapan
  2. 2.Intelligent Systems Research InstituteNational Institute of Advanced Industrial Science and Technology (AIST)TsukubaJapan

Personalised recommendations