Autonomous 3D Model Generation of Unknown Objects for Dual-Manipulator Humanoid Robots

  • Adrian LlopartEmail author
  • Ole Ravn
  • Nils A. Andersen
  • Jong-Hwan Kim
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 751)


This paper proposes a novel approach for the autonomous 3D model generation of unknown objects. A humanoid robot (or any setup with two manipulators) holds the object to model in one hand, views it from different perspectives and registers the depth information using a RGB-D sensor. The occlusions due to limited movement of the manipulator and the gripper itself covering the object are avoided by switching the object from one hand to the other. This allows for additional viewpoints leading to the registration of more depth information of the object. The contributions of this paper are as follows: 1. A humanoid robot that manipulates objects and obtains depth information 2. Tracing the hand movements with the robots head to be able to see the object at every moment 3. Filtering the point clouds to remove parts of the robot from them 4. Utilizing the Normal Iterative Closest Point algorithm (depth points, surface normals and curvature information) to register point clouds over time. This method will be applied to those pointclouds that include the robots gripper for optimal convergence; the resultant transform is then applied to those point clouds that describe only the segmented object 5. Changing the object from one hand to another 6. Merging the resulting object’s partial point clouds from both the left and right hands 7. Generating a mesh of the object based on the triangulation of final points of the object’s surface. No prior knowledge of the objects is necessary. No human intervention nor external help (i.e visual markers, turntables ...) is required either.


Humanoid robot 3D model creation Point cloud processing 


  1. 1.
    Aldoma, A., Tombari, F., Stefano, L.D., Vincze, M.: A global hypotheses verification method for 3D object recognition. In: 2012, European Conference on Computer Vision (ECCV). Lecture Notes in Computer Science, vol. 7574. SpringerGoogle Scholar
  2. 2.
    Zhu, M., Derpanis, K., Yang, Y., Brahmbhatt, S., Zhang, M., Phillips, C., Lecce, M., Daniilidis, K.: Single image 3D object detection and pose estimation for grasping. In: Proceedings International Conference on Robotics and Automation (ICRA), Hong Kong, China, 31 May–5 June 2014Google Scholar
  3. 3.
    Sarkar, K., Varanasi1, K., Stricker, D.: Trained 3D models for CNN based object recognition. In: Proceedings International Conference on Computer Vision (ICCV), Santiago, Chile, 13–16 Dec 2015Google Scholar
  4. 4.
    Peng, X., Sun, B., Ali, K., Saenko, K.: Learning deep object detectors from 3D models. In: Proceedings International Conference on Computer Vision (ICCV), Santiago, Chile, 13–16 Dec 2015Google Scholar
  5. 5.
    Gupta, S., Arbelaez, P., Girshick, R., Malik, J.: Aligning 3D models to RGB-D images of cluttered scenes. In: Proceedings International Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015Google Scholar
  6. 6.
    Pas, A., Platt, R.: Using geometry to detect grasp poses in 3D point clouds. In: Proceedings International Symposium on Robotics Research (ISRR), Genova, Italy, Sept 2015Google Scholar
  7. 7.
    Denkowski, M.: GPU accelerated 3D object reconstruction. Procedia Comput. Sci. 18, 290–298 (2013)Google Scholar
  8. 8.
    Mihalyi, R.-G., Pathak, K., Vaskevicius, N., Fromm, T., Birk, A.: Robust 3d object modeling with a low-cost RGBD-sensor and AR-markers for applications with untrained end-users. Robot. Auto. Syst. 66, 1–17 (2015)Google Scholar
  9. 9.
    Xie, J., Hsu, Y.-F., Feris, R., Sun, M.-T.: Fine registration of 3d point clouds fusing structural and photometric information using an RGB-D camera. In: J. Visual Commun. Image Represent. 32, 194–204 (2015)Google Scholar
  10. 10.
    Foissotte, T., Stasse, O., Escande, A., Wieber, P.-B., Kheddar, A.: A two-steps next-best-view algorithm for autonomous 3D object modeling by a humanoid robot. In: Proceedings International Conference on Robotics and Automation (ICRA), Kobe, Japan, 12–17 May 2009Google Scholar
  11. 11.
    Jaiswal, M., Xie, J., Sun, M.-T.: 3D object modeling with a Kinect camera. In: Proceedings Signal and Information Processing Association Annual Summit and Conference (APSIPA), Chiang Mai, Thailand, 6–9 June 2014Google Scholar
  12. 12.
    Krainin, M., Henry, P., Ren, X.: Manipulator and object tracking for in-hand 3d object modeling. Int. J. Robot. Res.Google Scholar
  13. 13.
    Llopart, A., Ravn, O., Andersen, N., Kim, J.-H.: Generalized framework for the parallel semantic segmentation of multiple objects and posterior manipulation. In: Proceedings International Conference on Robotics and Biomimetics (ROBIO), Macau, China, 5–8 Dec 2017Google Scholar
  14. 14.
    Rusu, R., Marton, Z.C., Blodow, N., Dolha, M., Beetz, M.: Towards 3d point cloud based object maps for household environments. Robot. Auto. Syst. 56(11), 927–941 (2008)Google Scholar
  15. 15.
    Marton, C., Radu, R., Beetz, M.: On fast surface reconstruction methods for large and noisy datasets. In: Proceedings International Conference on Robotics and Automation (ICRA), Kobe, Japan, 12–17 May 2009Google Scholar
  16. 16.
    Weise, T., Wismer, T., Leibe, B., Gool, L.V.: In-hand scanning with online loop closure. In: Proceedings International Conference on Computer Vision Workshops (ICCV Workshops), Kyoto, Japan, 27 Sept–4 Oct 2009Google Scholar
  17. 17.
    Krainin, M., Curless, B., Fox, D.: Autonomous generation of complete 3D object models using next best view manipulation planning. In: Proceedings International Conference on Robotics and Automation (ICRA), Shanghai, China, 9–13 May 2011Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2019

Authors and Affiliations

  • Adrian Llopart
    • 1
    • 2
    Email author
  • Ole Ravn
    • 1
  • Nils A. Andersen
    • 1
  • Jong-Hwan Kim
    • 2
  1. 1.AUT Group, Department of Electrical EngineeringDTUKgs. LyngbyDenmark
  2. 2.RIT Lab, School of Electrical EngineeringKAISTDaejeonRepublic of Korea

Personalised recommendations