Real-Time RGB-D Mapping and 3-D Modeling on the GPU Using the Random Ball Cover

  • Sebastian Bauer
  • Jakob Wasza
  • Felix Lugauer
  • Dominik Neumann
  • Joachim Hornegger


In this chapter, we present a system for real-time point cloud mapping and scene reconstruction based on an efficient implementation of the iterative closest point (ICP) algorithm on the graphics processing unit (GPU). Compared to state-of-the-art approaches that achieve real-time performance using projective data association schemes which operate on the 3-D scene geometry solely, our method allows to incorporate additional complementary information to guide the registration process. In this work, the ICP’s nearest neighbor search evaluates both geometric and photometric information in a direct manner, achieving robust mappings in real-time. In order to overcome the performance bottleneck in nearest neighbor search space traversal, we exploit the inherent computation parallelism of GPUs. In particular, we have adapted the random ball cover (RBC) data structure and search algorithm, originally proposed for high-dimensional problems, to low-dimensional RGB-D data. The system is validated on scene and object reconstruction scenarios. Our implementation achieves frame-to-frame registration runtimes of less than 20 ms on an off-the-shelf consumer GPU.


  1. 1.
    Akenine-Möller, T., Haines, E., Hoffman, N.: Real-Time Rendering, 3rd edn. AK Peters, Natick, MA (2008) CrossRefGoogle Scholar
  2. 2.
    Bailey, T., Durrant-Whyte, H.: Simultaneous localization and mapping (SLAM): part II. IEEE Robot. Autom. Mag. 13(3), 108–117 (2006) CrossRefGoogle Scholar
  3. 3.
    Bauer, S., Wasza, J., Haase, S., Marosi, N., Hornegger, J.: Multi-modal surface registration for markerless initial patient setup in radiation therapy using Microsoft’s Kinect sensor. In: International Conference on Computer Vision—Workshop on Consumer Depth Cameras for Computer Vision, pp. 1175–1181 (2011) Google Scholar
  4. 4.
    Besl, P., McKay, N.: A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992) CrossRefGoogle Scholar
  5. 5.
    Blais, G., Levine, D.M.: Registering multiview range data to create 3-D computer objects. IEEE Trans. Pattern Anal. Mach. Intell. 17(8), 820–824 (1995) CrossRefGoogle Scholar
  6. 6.
    Castaneda, V., Mateus, D., Navab, N.: SLAM combining ToF and high-resolution cameras. In: IEEE Workshop on Applications of Computer Vision, pp. 672–678 (2011) CrossRefGoogle Scholar
  7. 7.
    Cayton, L.: A nearest neighbor data structure for graphics hardware. In: International Workshop on Accelerating Data Management Systems Using Modern Processor and Storage Architectures (2010) Google Scholar
  8. 8.
    Cayton, L.: Accelerating nearest neighbor search on manycore systems. CoRR arXiv:1103.2635 (2011)
  9. 9.
    Chen, Y., Medioni, G.: Object modelling by registration of multiple range images. Image Vis. Comput. 10(3), 145–155 (1992) CrossRefGoogle Scholar
  10. 10.
    Curless, B., Levoy, M.: A volumetric method for building complex models from range images. In: Conference on Computer Graphics and Interactive Techniques, SIGGRAPH, pp. 303–312. ACM, New York (1996) Google Scholar
  11. 11.
    Druon, S., Aldon, M., Crosnier, A.: Color constrained ICP for registration of large unstructured 3D color data sets. In: IEEE International Conference on Information Acquisition, pp. 249–255 (2006) CrossRefGoogle Scholar
  12. 12.
    Engelhard, N., Endres, F., Hess, J., Sturm, J., Burgard, W.: Real-time 3D visual SLAM with a hand-held RGB-D camera. In: RGB-D Workshop on 3D Perception in Robotics, European Robotics Forum (2011) Google Scholar
  13. 13.
    Fioraio, N., Konolige, K.: Realtime visual and point cloud SLAM. In: RGB-D Workshop: Advanced Reasoning with Depth Cameras, Robotics Science and Systems Conference (2011) Google Scholar
  14. 14.
    Garcia, J., Zalevsky, Z.: Range mapping using speckle decorrelation. US Patent No. 7433024 (2008) Google Scholar
  15. 15.
    Garcia, V., Debreuve, E., Barlaud, M.: Fast k nearest neighbor search using GPU. In: IEEE Conference on Computer Vision and Pattern Recognition—Workshop on Computer Vision on GPU (2008) Google Scholar
  16. 16.
    Gevers, T., Smeulders, A.W.: Color-based object recognition. Pattern Recognit. 32(3), 453–464 (1999) CrossRefGoogle Scholar
  17. 17.
    Harris, M., Sengupta, S., Owens, J.D.: Parallel prefix sum (scan) with CUDA. In: GPU Gems 3, pp. 851–876. Addison-Wesley, Reading (2007) Google Scholar
  18. 18.
    He, K., Sun, J., Tang, X.: Guided image filtering. In: European Conference on Computer Vision, pp. 1–14 (2010) Google Scholar
  19. 19.
    Henry, P., Krainin, M., Herbst, E., Ren, X., Fox, D.: RGB-D mapping: using depth cameras for dense 3D modeling of indoor environments. In: International Symposium on Experimental Robotics (2010) Google Scholar
  20. 20.
    Hoberock, J., Bell, N.: Thrust: a parallel template library (2010). URL Version 1.3.0
  21. 21.
    Horn, B.: Closed-form solution of absolute orientation using unit quaternions. J. Opt. Soc. Am. 4(4), 629–642 (1987) MathSciNetCrossRefGoogle Scholar
  22. 22.
    Huhle, B., Jenke, P., Strasser, W.: On-the-fly scene acquisition with a handy multi-sensor system. Int. J. Intell. Syst. Technol. Appl. 5, 255–263 (2008) Google Scholar
  23. 23.
    Izadi, S., Newcombe, R.A., Kim, D., Hilliges, O., Molyneaux, D., Hodges, S., Kohli, P., Shotton, J., Davison, A.J., Fitzgibbon, A.W.: KinectFusion: real-time dynamic 3D surface reconstruction and interaction. In: ACM Symposium on User Interface Software and Technology, p. 23 (2011) Google Scholar
  24. 24.
    Johnson, A., Kang, S.B.: Registration and integration of textured 3-D data. In: International Conference on Recent Advances in 3-D Digital Imaging and Modeling, pp. 234–241 (1997) Google Scholar
  25. 25.
    Joung, J.H., An, K.H., Kang, J.W., Chung, M.J., Yu, W.: 3D environment reconstruction using modified color ICP algorithm by fusion of a camera and a 3D laser range finder. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3082–3088 (2009) Google Scholar
  26. 26.
    Korn, G.A., Korn, T.M.: Mathematical Handbook for Scientists and Engineers: Definitions, Theorems, and Formulas for Reference and Review. Dover, New York (2000) Google Scholar
  27. 27.
    Loop, C., Blinn, J.: Real-time GPU rendering of piecewise algebraic surfaces. ACM Trans. Graph. 25(3), 664–670 (2006) CrossRefGoogle Scholar
  28. 28.
    May, S., Droeschel, D., Holz, D., Fuchs, S., Malis, E., Nüchter, A., Hertzberg, J.: Three-dimensional mapping with time-of-flight cameras. J. Field Robot. 26, 934–965 (2009) CrossRefGoogle Scholar
  29. 29.
    McGuire, M.: A fast, small-radius GPU median filter. In: ShaderX6, pp. 165–173. Charles River Media (2008) Google Scholar
  30. 30.
    Neumann, D., Lugauer, F., Bauer, S., Wasza, J., Hornegger, J.: Real-time RGB-D mapping and 3-D modeling on the GPU using the random ball cover data structure. In: International Conference on Computer Vision—Workshop on Consumer Depth Cameras for Computer Vision, pp. 1161–1167 (2011) Google Scholar
  31. 31.
    Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohli, P., Shotton, J., Hodges, S., Fitzgibbon, A.W.: KinectFusion: real-time dense surface mapping and tracking. In: IEEE International Symposium on Mixed and Augmented Reality, pp. 127–136 (2011) CrossRefGoogle Scholar
  32. 32.
    Nüchter, A., Surmann, H., Lingemann, K., Hertzberg, J., Thrun, S.: 6D SLAM with an application in autonomous mine mapping. In: IEEE International Conference on Robotics and Automation, vol. 2, pp. 1998–2003 (2004) Google Scholar
  33. 33.
    Qiu, D., May, S., Nüchter, A.: GPU-accelerated nearest neighbor search for 3D registration. In: International Conference on Computer Vision Systems, pp. 194–203. Springer, Berlin (2009) CrossRefGoogle Scholar
  34. 34.
    Reis, G., Zeilfelder, F., Hering-Bertram, M., Farin, G.E., Hagen, H.: High-quality rendering of quartic spline surfaces on the GPU. IEEE Trans. Vis. Comput. Graph. 14(5), 1126–1139 (2008) CrossRefGoogle Scholar
  35. 35.
    Rusinkiewicz, S., Hall-Holt, O., Levoy, M.: Real-time 3D model acquisition. ACM Trans. Graph. 21(3), 438–446 (2002) CrossRefGoogle Scholar
  36. 36.
    Rusinkiewicz, S., Levoy, M.: Efficient variants of the ICP algorithm. In: International Conference on 3-D Digital Imaging and Modeling, pp. 145–152 (2001) CrossRefGoogle Scholar
  37. 37.
    Wasza, J., Bauer, S., Haase, S., Hornegger, J.: Real-time preprocessing for dense 3-D range imaging on the GPU: defect interpolation, bilateral temporal averaging and guided filtering. In: International Conference on Computer Vision—Workshop on Consumer Depth Cameras for Computer Vision, pp. 1221–1227 (2011) Google Scholar

Copyright information

© Springer-Verlag London 2013

Authors and Affiliations

  • Sebastian Bauer
    • 1
  • Jakob Wasza
    • 1
  • Felix Lugauer
    • 1
  • Dominik Neumann
    • 1
  • Joachim Hornegger
    • 2
  1. 1.Pattern Recognition Lab, Department of Computer ScienceFriedrich-Alexander-Universität Erlangen-NürnbergErlangenGermany
  2. 2.Erlangen Graduate School in Advanced Optical Technologies (SAOT) & Pattern Recognition Lab, Department of Computer ScienceFriedrich-Alexander-Universität Erlangen-NürnbergErlangenGermany

Personalised recommendations