An Adaptive Robotic System for Doing Pick and Place Operations with Deformable Objects

  • Troels Bo Jørgensen
  • Sebastian Hoppe Nesgaard Jensen
  • Henrik Aanæs
  • Niels Worsøe Hansen
  • Norbert Krüger


This paper presents a robot system for performing pick and place operations with deformable objects. The system uses a structured light scanner to capture a point cloud of the object to be grasped. This point cloud is then analyzed to determine a pick and place action. Finally, the determined action is executed by the robot to solve the task. The robotic placement strategy contains several free parameters, which should be chosen in a context-specific manner. To determine these parameters we rely on simulation-based optimization of the individual use cases. The entire system is tested extensively in real world trials. First, the reliability of the grasp is evaluated for 7 different types of pork cuts. Then the validity of the simulation-based optimization of the placement strategy is evaluated for 2 of the most different pork cuts, to show the generality of the overall approach.


Robotic manipulation Deformable objects Structured light scanner Vision-based meat analysis Simulation-based optimization 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



The financial support from the The Danish Innovation Foundation through the strategic platform “MADE-Platform for Future Production” and from the EU project ReconCell (FP7-ICT-680431) is gratefully acknowledged.


  1. 1.
    Balaguer, B., Carpin, S.: Combining imitation and reinforcement learning to fold deformable planar objects. In: IROS, pp. 1405–1412. IEEE. (2011)
  2. 2.
    Berkenkamp, F., Krause, A., Schoellig, A.P.: Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics. arXiv:1602.04450 (2016)
  3. 3.
    Bodenhagen, L., Fugl, A.R., Jordt, A., Willatzen, M., Andersen, K.A., Olsen, M.M., Koch, R., Petersen, H.G., Krüger, N.: An adaptable robot vision system performing manipulation actions with flexible objects. IEEE Trans. Autom. Sci. Eng. 11(3), 749–765 (2014)CrossRefGoogle Scholar
  4. 4.
    Buch, J.P., Laursen, J.S., Sørensen, L.C., Ellekilde, L.P., Kraft, D., Schultz, U.P., Petersen, H.G.: Applying simulation and a domain-specific language for an adaptive action library. In: International Conference on Simulation, Modeling, and Programming for Autonomous Robots, pp. 86–97. Springer (2014)Google Scholar
  5. 5.
    Caelles, S., Maninis, K.K., Pont-Tuset, J., Leal-Taixé, L., Cremers, D., Van Gool, L.: One-shot video object segmentation. In: Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  6. 6.
    Calandra, R., Seyfarth, A., Peters, J., Deisenroth, M.P.: Bayesian optimization for learning gaits under uncertainty. Ann. Math. Artif. Intell. 76(1-2), 5–23 (2016)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Costa, A., Nannicini, G.: Rbfopt: an open-source library for black-box optimization with costly function evaluations. Optimization Online (4538) (2014)Google Scholar
  8. 8.
    Eberly, D.: Thin plate splines. Geometric Tools Inc 2002, 116 (2002)Google Scholar
  9. 9.
    Eiríksson, E.R., Wilm, J., Pedersen, D.B., Aanæs, H.: Precision and accuracy parameters in structured light 3-d scanning. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences 40 (2015)Google Scholar
  10. 10.
    Ellekilde, L.P., Jorgensen, J.A.: Robwork: A flexible toolbox for robotics research and education. In: 2010 41st International Symposium on and 2010 6th German Conference on Robotics of Robotics (ISR), (ROBOTIK), pp. 1–7. VDE (2010)Google Scholar
  11. 11.
    Geng, J.: Structured-light 3d surface imaging: a tutorial. Adv. Opt. Photon. 3(2), 128–160 (2011)CrossRefGoogle Scholar
  12. 12.
    Gonzalez-Jorge, H., Rodríguez-Gonzálvez, P., Martínez-Sánchez, J., González-Aguilera, D., Arias, P., Gesto, M., Díaz-Vilariño, L.: Metrological comparison between kinect i and kinect ii sensors. Measurement 70, 21–26 (2015)CrossRefGoogle Scholar
  13. 13.
    Gupta, M., Nayar, S.K.: Micro phase shifting. Proc. IEEE CVPR, pp. 813–820 (2012)Google Scholar
  14. 14.
    Gupta, M., Yin, Q., Nayar, S.K.: Structured light in sunlight. In: The IEEE International Conference on Computer Vision (ICCV) (2013)Google Scholar
  15. 15.
    Hemker, T., Stelzer, M., von Stryk, O., Sakamoto, H.: Efficient walking speed optimization of a humanoid robot. Int. J. Robot. Res. 28(2), 303–314 (2009)CrossRefGoogle Scholar
  16. 16.
    Jensen, S., Wilm, J., Aanæs, cH.: An error analysis of structured light scanning of biological tissue, pp. 135–145 Springer (2017)Google Scholar
  17. 17.
    Jørgensen, T.B., Holm, P.H.S., Petersen, H.G., Krüger, N.: Intelligent Robotics and Applications: 8th International Conference, ICIRA 2015, Portsmouth, UK, August 24-27, 2015. Springer International Publishing, Cham (2015)Google Scholar
  18. 18.
    Jørgensen, T.B., Debrabant, K., Krüger, N.: Robust optimization of robotic pick and place operations for deformable objects through simulation. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 3863–3870 (2016)Google Scholar
  19. 19.
    Jørgensen, T.B., Pedersen, M.M., Hansen, N.W., Hansen, B.R., Kruger, N.: A flexible suction based grasp tool and associated grasp strategies for handling meat. International Conference on Mechatronics and Robotics Engineering accepted (2017)Google Scholar
  20. 20.
    Jørgensen, T.B., Wolniakowski, A., Petersen, H.G., Debrabant, K., Kruger, N.: Robust optimization with applications to design of context specific robot solutions. Robotics and Computer Integrated Manufacturing Submitted (2017)Google Scholar
  21. 21.
    Kabsch, W.: A solution for the best rotation to relate two sets of vectors. Acta Crystallogr., Sect. A: Cryst. Phys., Diffr., Theor. Gen. Crystallogr. 32(5), 922–923 (1976)CrossRefGoogle Scholar
  22. 22.
    Koh, Y.J., Kim, C.S.: Primary object segmentation in videos based on region augmentation and reduction. (2017)
  23. 23.
    Kruse, D., Radke, R.J., Wen, J.T.: Human-robot collaborative handling of highly deformable materials. In: American Control Conference (ACC), 2017, pp. 1511–1516. IEEE (2017)Google Scholar
  24. 24.
    Li, Y., Wang, Y., Case, M., Chang, S.F., Allen, P.K.: Real-time pose estimation of deformable objects using a volumetric approach. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2014), pp. 1046–1052. IEEE (2014)Google Scholar
  25. 25.
    Long, P., Khalil, W., Martinet, P.: Robotic deformable object cutting: from simulation to experimental validation (2014)Google Scholar
  26. 26.
    Loshchilov, I., Schoenauer, M., Sebag, M.: Adaptive coordinate descent (2011)Google Scholar
  27. 27.
    Mesit, J., Guha, R., Chaudhry, S.: 3d soft body simulation using mass-spring system with internal pressure force and simplified implicit integration. J. Comput. 2(8), 34–43 (2007)CrossRefGoogle Scholar
  28. 28.
    Misimi, E., Øye, E.R., Eilertsen, A., Mathiassen, J.R., AAsebø, O.B., Gjerstad, T., Buljo, J., Skotheim, Ø.: Gribbot–robotic 3d vision-guided harvesting of chicken fillets. Comput. Electron. Agric. 121, 84–100 (2016)CrossRefGoogle Scholar
  29. 29.
    Nabil, E., Belhassen-Chedli, B., Grigore, G.: Soft material modeling for robotic task formulation and control in the muscle separation process. Robot. Comput. Integr. Manuf. 32, 37–53 (2015)CrossRefGoogle Scholar
  30. 30.
    Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine-Hornung, A., Van Gool, L.: The 2017 davis challenge on video object segmentation. arXiv:1704.00675 (2017)
  31. 31.
    Posdamer, J., Altschuler, M.: Surface measurement by space-encoded projected beam systems. Comput. Graphics Image Process. 18, 1–17 (1982). CrossRefGoogle Scholar
  32. 32.
    Rusu, R.B., Cousins, S.: 3D is here: Point Cloud Library (PCL). In: IEEE International Conference on Robotics and Automation (ICRA). Shanghai, China (2011)Google Scholar
  33. 33.
    Rusu, R.B., Cousins, S.: Region growing segmentation. (2017)
  34. 34.
    Schulman, J., Lee, A., Ho, J., Abbeel, P.: Tracking deformable objects with point clouds. In: 2013 IEEE International Conference on Robotics and Automation (ICRA), pp. 1130–1137. IEEE (2013)Google Scholar
  35. 35.
    Tappen, M.F., Freeman, W.T.: Comparison of graph cuts with belief propagation for stereo, using identical mrf parameters. In: Null, p. 900. IEEE (2003)Google Scholar
  36. 36.
    Tesch, M., Schneider, J., Choset, H.: Using response surfaces and expected improvement to optimize snake robot gait parameters. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1069–1074. IEEE (2011)Google Scholar
  37. 37.
    Voigtlaender, P., Leibe, B.: Online adaptation of convolutional neural networks for video object segmentation. In: BMVC (2017)Google Scholar
  38. 38.
    Wilm, J., Olesen, O.V., Larsen, R.: Slstudio: open-source framework for real-time structured light. Proceedings of the 4th International Conference on Image Processing Theory, Tools and Application (ipta 2014) p. 7002001. (2014)
  39. 39.
    Wolniakowski, A., Jorgensen, J.A., Miatliuk, K., Petersen, H.G., Kruger, N.: Task and context sensitive optimization of gripper design using dynamic grasp simulation. In: 2015 20th International Conference on Methods and Models in Automation and Robotics (MMAR), pp. 29–34. IEEE (2015)Google Scholar
  40. 40.
    Zoumponos, G.T., Aspragathos, N.A.: A fuzzy strategy for the robotic folding of fabrics with machine vision feedback. Industrial Robot: An International Journal 37(3), 302–308 (2010)CrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2018

Authors and Affiliations

  1. 1.Maersk McKinney Møller InstituteUniversity of Southern DenmarkOdense MDenmark
  2. 2.DTU ComputeTechnical University of DenmarkKongens LyngbyDenmark
  3. 3.Danish Meat Research InstituteDanish Technological InstituteTaastrupDenmark

Personalised recommendations