R\(^{3}\)P: Real-time RGB-D Registration Pipeline

  • Hani Javan HemmatEmail author
  • Egor Bondarev
  • Peter H. N. de With
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10016)


Applications based on colored 3-D data sequences suffer from lack of efficient algorithms for transformation estimation and key points extraction to perform accurate registration and sensor localization either in the 2-D or 3-D domain. Therefore, we propose a real-time RGB-D registration pipeline, named R3P, presented in processing layers. In this paper, we present an evaluation of several algorithm combinations for each layer, to optimize the registration and sensor localization for specific applications. The resulting dynamic reconfigurability of R3P makes it suitable as a front-end system for any SLAM reconstruction algorithm. Evaluation results on several public datasets reveal that R3P delivers real-time registration with 59 fps and high accuracy with the relative pose error (for a time span of 40 frames) for rotation and translation of \(0.5^\circ \) and 8 mm, respectively. All the heterogeneous dataset and implementations are publicly available under an open-source license [21].


RGB-D registration Real-time Heterogeneous 3-D Reconstruction Simultaneously Localization and Mapping (SLAM) 


  1. 1.
    Javan Hemmat, H., et al.: Real-time planar segmentation of depth images: from 3D edges to segmented planes. J. Electron. Imaging 24(5), 051008 (2015)CrossRefGoogle Scholar
  2. 2.
    Mur-Artal, R., Montiel, J.M.M., Tards, J.D.: ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Trans. Robot. 31(5), 1147–1163 (2015)CrossRefGoogle Scholar
  3. 3.
    Lu, Y., Song, D.: Visual navigation using heterogeneous landmarks and unsupervised geometric constraints. IEEE Trans. Robot. 31(3), 736–749 (2015)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Bouaziz, S., Pauly, M.: Dynamic 2D/3D registration for the kinect. In: ACM SIGGRAPH Courses, SIGGRAPH 2013, pp. 21:1–21:14. ACM, New York (2013)Google Scholar
  5. 5.
    Henry, P., et al.: RGB-D mapping: using kinect-style depth cameras for dense 3D modeling of indoor environments. Int. J. Robot. Res. 31(5), 647–663 (2012)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Newcombe, R.A., Lovegrove, S., Davison, A.: DTAM: dense tracking and mapping in real-time. In: IEEE International Conference on Computer Vision, pp. 2320–2327 (2011)Google Scholar
  7. 7.
    Labbe, M., Michaud, F.: Appearance-based loop closure detection for online large-scale and long-term operation. IEEE Trans. Robot. 29(3), 734–745 (2013)CrossRefGoogle Scholar
  8. 8.
    Endres, F., Hess, J., Sturm, J., Cremers, D., Burgard, W.: 3-D mapping with an RGB-D camera. IEEE Trans. Robot. 30(1), 177–187 (2014)CrossRefGoogle Scholar
  9. 9.
    Hu, G., Huang, S., Zhao, L., Alempijevic, A., Dissanayake, G.: A robust RGB-D slam algorithm. In: International Conference on Intelligent Robots and Systems (IROS), pp. 1714–1719 (2012)Google Scholar
  10. 10.
    Andreasson, H., Stoyanov, T.: Real time registration of RGB-D data using local visual features and 3D-NDT registration. In: International Conference on Robotics and Automation (2012)Google Scholar
  11. 11.
    Stckler, J., Behnke, S.: Model learning and real-time tracking using multiresolution surfel maps. In: Proceedings of the AAAI Conference on Artificial Intelligence (2012)Google Scholar
  12. 12.
    Steinbrucker, F., Sturm, J., Cremers, D.: Real-time visual odometry from dense RGB-D images. In: ICCV Workshops, pp. 719–722. IEEE (2011)Google Scholar
  13. 13.
    Dib, A., et al.: A real time visual slam for RGB-D cameras based on chamfer distance and occupancy grid. In: International Conference on Advanced Intelligent Mechatronics, pp. 652–657 (2014)Google Scholar
  14. 14.
    Kerl, C., Sturm, J., Cremers, D.: Robust odometry estimation for RGB-D cameras. In: ICRA (2013)Google Scholar
  15. 15.
    Choi, C., Trevor, A.J.B., Christensen, H.I.: RGB-D edge detection and edge-based registration. In: IROS, pp. 1568–1575. IEEE (2013)Google Scholar
  16. 16.
    Lu, Y., Song, D.: Robust RGB-D odometry using point and line features. In: IEEE International Conference on Computer Vision, ICCV (2015)Google Scholar
  17. 17.
    Morell-Gimenez, V., et al.: A comparative study of registration methods for RGB-D video of static scenes. Sensors 14(5), 8547 (2014)CrossRefGoogle Scholar
  18. 18.
    Seifi, S., et al.: DeReEs: real-time registration of RGB-D images using image-based feature detection and robust 3D correspondence estimation and refinement. In: 29th International Conference on Image and Vision Computing, pp. 136–141 (2014)Google Scholar
  19. 19.
    Yousif, K., et al.: A real-time RGB-D registration and mapping approach by heuristically switching between photometric and geometric information. In: International Conference on Information Fusion (2014)Google Scholar
  20. 20.
    Figueroa, N., Dong, H., Saddik, A.E.: A combined approach toward consistent reconstructions of indoor spaces based on 6D RGB-D odometry and kinect fusion. ACM Trans. Intell. Syst. Technol. 6(2), 14:1–14:10 (2015)CrossRefGoogle Scholar
  21. 21.
    R3P-related codes, docs and dataset. Accessed 9 Sept 2016
  22. 22.
    Bradski, G.: OpenCV: an open source computer vision library. Dr. Dobbs Journal of Software Tools (2000)Google Scholar
  23. 23.
    Rusu, R., Cousins, S.: 3D is here: point cloud library (PCL). In: IEEE International Conference on Robotics and Automation (ICRA), pp. 1–4 (2011)Google Scholar
  24. 24.
    Handa, A., Whelan, T., McDonald, J., Davison, A.: A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In: IEEE ICRA (2014)Google Scholar
  25. 25.
    Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A benchmark for the evaluation of RGB-D slam systems. In: Intelligent Robot Systems (IROS) (2012)Google Scholar
  26. 26.
    Shotton, J., et al.: Scene coordinate regression forests for camera relocalization in RGB-D images. In: Computer Vision and Pattern Recognition (IEEE CVPR) (2013)Google Scholar
  27. 27.
    Javan Hemmat, H., Bondarev, E., Dubbelman, G., de With, P.H.N.: Improved ICP-based pose estimation by distance-aware 3D mapping. In: 9th International Conference on Computer Vision Theory and Applications, pp. 360–367 (2014)Google Scholar
  28. 28.
    Javan Hemmat, H., Bondarev, E., de With, P.H.N.: Exploring distance-aware weighting strategies for accurate reconstruction of voxel-based 3D synthetic models. In: Gurrin, C., Hopfgartner, F., Hurst, W., Johansen, H., Lee, H., O’Connor, N. (eds.) MMM 2014. LNCS, vol. 8325, pp. 412–423. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-04114-8_35 CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Hani Javan Hemmat
    • 1
    Email author
  • Egor Bondarev
    • 1
  • Peter H. N. de With
    • 1
  1. 1.Eindhoven University of TechnologyEindhovenNetherlands

Personalised recommendations