Machine Vision and Applications

, Volume 27, Issue 3, pp 377–385 | Cite as

Realistic surface geometry reconstruction using a hand-held RGB-D camera

Original Paper


In this paper, we have proposed a novel approach for the reconstruction of real object/scene with realistic surface geometry using a hand-held, low-cost, RGB-D camera. To achieve accurate reconstruction, the most important issues to consider are the quality of the geometry information provided and the global alignment method between frames. In our approach, new surface geometry refinement is used to recover finer scale surface geometry from depth data by utilizing high-quality RGB images. In addition, a weighted multi-scale iterative closest point method is exploited to align each scan to the global model accurately. We show the effectiveness of the proposed surface geometry refinement method by comparing it with other depth refinement methods. We also show both the qualitative and quantitative results of reconstructed models by comparing it with other reconstruction methods.


3D reconstruction Kinect RGB-D images SLAM Volumetric representation Real time 


  1. 1.
    Beeler, T., Bickel, B., Beardsley, P., Sumner, B., Gross, M.: High-quality single-shot capture of facial geometry. ACM Trans. Graph. 29(4), 40:1–40:9 (2010). doi:10.1145/1778765.1778777.
  2. 2.
    Besl, P., McKay, H.: A method for registration of 3-d shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992). doi:10.1109/34.121791 CrossRefGoogle Scholar
  3. 3.
    Chen, Y., Medioni, G.: Object modelling by registration of multiple range images. Image Vis. Comput. 10(3), 145–155 (1992). doi:10.1016/0262-8856(92)90066-C CrossRefGoogle Scholar
  4. 4.
    Cui, Y., Schuon, S., Chan, D., Thrun, S., Theobalt, C.: 3d shape scanning with a time-of-flight camera. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR), pp 1173 –1180 (2010). doi:10.1109/CVPR.2010.5540082
  5. 5.
    Ferstl, D., Reinbacher, C., Ranftl, R., Ruether, M., Bischof, H.: Image guided depth upsampling using anisotropic total generalized variation. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp 993–1000 (2013). doi:10.1109/ICCV.2013.127
  6. 6.
    Freedman SAMMAY Barak (2008) Depth mapping using projected patterns.
  7. 7.
    Garcia, F., Mirbach, B., Ottersten, B., Grandidier, F., Cuesta, A.: Pixel weighted average strategy for depth sensor data fusion. In: 2010 17th IEEE International Conference on Image Processing (ICIP), pp 2805–2808 (2010). doi:10.1109/ICIP.2010.5651112
  8. 8.
    Garcia, F., Aouada, D., Mirbach, B., Solignac, T., Ottersten, B.: A new multi-lateral filter for real-time depth enhancement. In: 2011 8th IEEE International Conference on Advanced Video and Signal-Based Surveillance (AVSS), pp 42 –47 (2011). doi:10.1109/AVSS.2011.6027291
  9. 9.
    Geiger, A., Ziegler, J., Stiller, C.: Stereoscan: dense 3d reconstruction in real-time. In: Intelligent Vehicles Symposium. IEEE, New York, pp 963–968 (2011)Google Scholar
  10. 10.
    Godin, G., Rioux, M., Baribeau, R.: Three-dimensional registration using range and intensity information. Proc. SPIE (1994). doi:10.1117/12.189139
  11. 11.
    He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013). doi:10.1109/TPAMI.2012.213 CrossRefGoogle Scholar
  12. 12.
    Herrera, C.D., Kannala, J., Heikkil, J.: Joint depth and color camera calibration with distortion correction. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 10, pp. 2058–2064 (2012). doi:10.1109/TPAMI.2012.125
  13. 13.
    Hirschmuller, H., Scharstein, D.: Evaluation of cost functions for stereo matching. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR ’07, pp 1–8 (2007). doi:10.1109/CVPR.2007.383248
  14. 14.
    Kazhdan, M., Hoppe, H.: Screened poisson surface reconstruction. ACM Trans. Graph. 32(3), 29:1–29:13 (2013). doi:10.1145/2487228.2487237
  15. 15.
    Khoshelham, K., Elberink, S.O.: Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12(2), 1437–1454 (2012). doi:10.3390/s120201437
  16. 16.
    Khoshelham, K., Dos Santos, D., Vosselman, G.: Generation and weighting of 3d point correspondences for improved registration of rgb-d data. In: ISPRS Annals of the Photogrammetry and Remote Sensing and Spatial Information Sciences, vol. II-5/W2 (2013)Google Scholar
  17. 17.
    Kopf, J., Cohen, M.F., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. In: ACM SIGGRAPH 2007 Papers. ACM, New York (2007). SIGGRAPH ’07. doi:10.1145/1275808.1276497
  18. 18.
    Lee, K.R., Nguyen, T.: Robust tracking and mapping with a handheld RGB-D camera. In: 2014 IEEE Winter Conference on Applications of Computer Vision (WACV), pp 1120–1127 (2014). doi:10.1109/WACV.2014.6835732
  19. 19.
    Lee, K.R., Khoshabeh, R., Nguyen, T.: Sampling-based robust multi-lateral filter for depth enhancement. In: 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO), pp 1124–1128 (2012)Google Scholar
  20. 20.
    Low, K.L.: Linear least-squares optimization for point-to-plane ICP surface registration. Tech. Rep. TR04-004. Department of Computer Science, University of North Carolina at Chapel Hill (2004)Google Scholar
  21. 21.
    Lu, J., Min, D., Pahwa, R., Do, M.: A revisit to mrf-based depth map super-resolution and enhancement. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 985–988 (2011). doi:10.1109/ICASSP.2011.5946571
  22. 22.
    Mohr, R., Quan, L., Veillon, F.: Relative 3d reconstruction using multiple uncalibrated images, pp. 543–548 (1995)Google Scholar
  23. 23.
    Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohli, P., Shotton, J., Hodges, S., Fitzgibbon, A.: Kinectfusion: real-time dense surface mapping and tracking. In: Proceedings of the 2011 10th IEEE International Symposium on Mixed and Augmented Reality. IEEE Computer Society, Washington, DC, ISMAR ’11, pp 127–136 (2011). doi:10.1109/ISMAR.2011.6092378
  24. 24.
    Nguyen, C., Izadi, S., Lovell, D.: Modeling kinect sensor noise for improved 3d reconstruction and tracking. In: 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), pp 524–530 (2012). doi:10.1109/3DIMPVT.2012.84
  25. 25.
    Oggier, T.: An all-solid-state optical range camera for 3d real-time imaging with sub-centimeter depth resolution (swissranger). Proc. SPIE 5249(65), 534–545 (2004)CrossRefGoogle Scholar
  26. 26.
    Park, J., Kim, H., Tai, Y.W., Brown, M., Kweon, I.: High quality depth map upsampling for 3d-tof cameras. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp 1623–1630 (2011). doi:10.1109/ICCV.2011.6126423
  27. 27.
    Pollefeys, M.: Self calibration and metric 3d reconstruction from uncalibrated image sequences. PhD thesis, Leuven (1999)Google Scholar
  28. 28.
    Rusinkiewicz, S., Levoy, M.: Efficient variants of the icp algorithm. In: International Conference on 3-D Digital Imaging and Modeling (2001)Google Scholar
  29. 29.
    Rusinkiewicz, S., Hall-Holt, O., Levoy, M.: Real-time 3d model acquisition. In: Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques. ACM, New York, SIGGRAPH ’02, pp. 438–446 (2002). doi:10.1145/566570.566600
  30. 30.
    Seitz, S., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 519–528 (2006). doi:10.1109/CVPR.2006.19
  31. 31.
    Slabaugh, G., Culbertson, B., Malzbender, T., Schafer, R.: A Survey of Methods for Volumetric Scene Reconstruction from Photographs. pp. 81–100 (2001).
  32. 32.
    Tenedorio, D., Fecho, M., Schwartzhaupt, J., Pardridge, R., Lue, J., Schulze, J.P.: Capturing geometry in real-time using a tracked microsoft kinect, pp. 82,890A–82,890A–14 (2012). doi:10.1117/12.912211
  33. 33.
    Tong, J., Zhou, J., Liu, L., Pan, Z., Yan, H.: Scanning 3d full human bodies using kinects. In: IEEE Transactions on Visualization and Computer Graphics 18, 643–650 (2012). doi:10.1109/TVCG.2012.56
  34. 34.
    Yang, Q., Yang, R., Davis, J., Nister, D.: Spatial-depth super resolution for range images. In: IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR ’07, pp 1–8 (2007). doi:10.1109/CVPR.2007.383211

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.Department of Electrical and Computer EngineeringUniversity of California San DiegoLa JollaUSA

Personalised recommendations