Journal of Real-Time Image Processing

, Volume 10, Issue 4, pp 647–666 | Cite as

Real-time 3D semi-local surface patch extraction using GPGPU

Application to 3D object recognition
  • Sergio Orts-EscolanoEmail author
  • Vicente Morell
  • Jose Garcia-Rodriguez
  • Miguel Cazorla
  • Robert B. Fisher
Special Issue Paper


Feature vectors can be anything from simple surface normals to more complex feature descriptors. Feature extraction is important to solve various computer vision problems: e.g. registration, object recognition and scene understanding. Most of these techniques cannot be computed online due to their complexity and the context where they are applied. Therefore, computing these features in real-time for many points in the scene is impossible. In this work, a hardware-based implementation of 3D feature extraction and 3D object recognition is proposed to accelerate these methods and therefore the entire pipeline of RGBD based computer vision systems where such features are typically used. The use of a GPU as a general purpose processor can achieve considerable speed-ups compared with a CPU implementation. In this work, advantageous results are obtained using the GPU to accelerate the computation of a 3D descriptor based on the calculation of 3D semi-local surface patches of partial views. This allows descriptor computation at several points of a scene in real-time. Benefits of the accelerated descriptor have been demonstrated in object recognition tasks. Source code will be made publicly available as contribution to the Open Source Point Cloud Library.


Real-time GPGPU RGB-D 3D local shape descriptor Object recognition 



This work was partially funded by the Valencian Government BEFPI/2012/056, and by the European Network of Excellence on High Performance and Embedded Architecture and Compilation (HiPEAC). Experiments were made possible with a generous donation of hardware from NVDIA.


  1. 1.
    Oikonomidis, I., Kyriazis, N., Argyros, A.A.: Full DoF tracking of a hand interacting with an object by modeling occlusions and physical constraints. In: Proceedings of the 2011 International Conference on Computer Vision. ICCV11, IEEE Computer Society, Washington DC, pp 2088–2095 (2011)Google Scholar
  2. 2.
    Izadi, S., Newcombe, R.A., Kim, D., Hilliges, O., Molyneaux, D., Hodges, S., Kohli, P., Shotton, J., Davison, A.J., Fitzgibbon, A.W.: Kinectfusion: real-time dynamic 3D surface reconstruction and interaction. In: SIGGRAPH Talks (2011)Google Scholar
  3. 3.
    Kerl, C., Sturm, J., Cremers, D.: Dense visual slam for RGB-D cameras. In: Proceedings of the International Conference on Intelligent Robot Systems (IROS) (2013)Google Scholar
  4. 4.
    Griffin, W., Wang, Y., Berrios, D., Olano, M.: Real-time GPU surface curvature estimation on deforming meshes and volumetric data sets. IEEE Trans. Vis. Comput. Graphics 18(10), 1603–1613 (2012)Google Scholar
  5. 5.
    Prisacariu, V., Reid, I.: fastHOG—a real-time GPU implementation of HOG. Technical Report 2310/09, Department of Engineering Science, Oxford University (2009)Google Scholar
  6. 6.
    Himmelsbach, M., Luettel, T., Wuensche, H.J.: Real-time object classification in 3D point clouds using point feature histograms. In: Proceedings of the 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems. IROS’09, IEEE Press, Piscataway, pp 994–1000 (2009)Google Scholar
  7. 7.
    Olesen, S., Lyder, S., Kraft, D., Krger, N., Jessen, J.: Real-time extraction of surface patches with associated uncertainties by means of kinect cameras. J. Real Time Image Process, 1–14 (2012)Google Scholar
  8. 8.
    Blum, M., Springenberg, J.T., Wülfing, J., Riedmiller, M.: A learned feature descriptor for object recognition in rgb-d data. In: ICRA, pp 1298–1303 (2012)Google Scholar
  9. 9.
    Lee, W., Park, N., Woo, W.: Depth-assisted real-time 3D object detection for augmented reality. ICAT’11, vol. 2, pp. 126–132 (2011)Google Scholar
  10. 10.
    Johnson, A.E., Hebert, M.: Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Trans. Pattern Anal. Mach. Intell. 21(5), 433–449 (1999) Google Scholar
  11. 11.
    Stein, F., Medioni, G.: Structural indexing: efficient 3D object recognition. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 125–145 (1992)Google Scholar
  12. 12.
    Chua, C.S., Jarvis, R.: Point signatures: a new representation for 3D object recognition. Int. J. Comput. Vis. 25(1), 63–85 (1997)Google Scholar
  13. 13.
    Ruiz-Correa, S., Shapiro, L., Melia, M.: A new signature-based method for efficient 3D object recognition. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1. CVPR 2001, pp. I-769–I-776 (2001)Google Scholar
  14. 14.
    Yamany, S.M., Farag, A.A.: Surface signatures: an orientation independent free-form surface representation scheme for the purpose of objects registration and matching. IEEE Trans. Pattern Anal. Mach. Intell. 24(8), 1105–1120 (2002)Google Scholar
  15. 15.
    Ashbrook, A., Fisher, R.: Finding surface correspondence for object recognition and registration. In: Proceedings of the 5th European Conference on Computer Vision, vol. II, Freiburg, pp. 674–680 (1998)Google Scholar
  16. 16.
    Chen, H., Bhanu, B.: 3D free-form object recognition in range images using local surface patches. Pattern Recognit. Lett. 28(10), 1252–1262 (2007)Google Scholar
  17. 17.
    Taati, B., Greenspan, M.: Local shape descriptor selection for object recognition in range data. Comput. Vis. Image Underst. 115(5), 681–694 (2011)Google Scholar
  18. 18.
    Shang, L., Greenspan, M.A.: Real-time object recognition in sparse range images using error surface embedding. Int. J.Comput. Vis. 89(2–3), 211–228 (2010)Google Scholar
  19. 19.
    Campbell, R.J., Flynn, P.J.: A survey of free-form object representation and recognition techniques. Comput. Vis. Image Underst. 81, 166–210 (2001)Google Scholar
  20. 20.
    Mian, A.S., Bennamoun, M., Owens, R.A.: Automatic correspondence for 3D modeling: an extensive review. Int. J. Shape Model. 11(2), 253–291 (2005)Google Scholar
  21. 21.
    Mian, A.S., Bennamoun, M., Owens, R.: Three-dimensional model-based object recognition and segmentation in cluttered scenes. IEEE Trans. Pattern Anal. Mach. Intell. 28(10), 1584–1601 (2006)Google Scholar
  22. 22.
    Mian, A.S., Bennamoun, M., Owens, R.A.: A novel representation and feature matching algorithm for automatic pairwise registration of range images. Int. J. Comput. Vis. 66(1), 19–40 (2006)Google Scholar
  23. 23.
    Hetzel, G., Leibe, B., Levi, P., Schiele, B.: 3D object recognition from range images using local feature histograms. In: CVPR (2). IEEE Computer Society, pp. 394–399 (2001)Google Scholar
  24. 24.
    Tombari, F., Salti, S., di Stefano, L.: Unique signatures of histograms for local surface description. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV (3). Volume 6313 of Lecture Notes in Computer Science. Springer, Berlin, pp. 356–369 (2010)Google Scholar
  25. 25.
    Khoshelham, K., Elberink, S.O.: Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12(2), 1437–1454 (2012)Google Scholar
  26. 26.
    NVIDIA. CUDA Programming Guide 5.0. (2012)Google Scholar
  27. 27.
    Rusu, R.B., Cousins, S.: 3D is here: Point Cloud Library (PCL). In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Shanghai (2011)Google Scholar
  28. 28.
    Zhang, Z.: Microsoft kinect sensor and its effect. IEEE MultiMedia 19(2), 4–10 (2012)Google Scholar
  29. 29.
    Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: Proceedings of the Sixth International Conference on Computer Vision. ICCV ’98, IEEE Computer Society, Washington, DC, pp. 839–846 (1998)Google Scholar
  30. 30.
    Chan, D., Buisman, H., Theobalt, C., Thrun, S.: A noise-aware filter for real-time depth upsampling. In: Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications—M2SFA2 2008, Marseille (2008)Google Scholar
  31. 31.
    Wasza, J., Bauer, S., Hornegger, J.: Real-time preprocessing for dense 3D range imaging on the GPU: defect interpolation, bilateral temporal averaging and guided filtering. In: ICCV Workshops, pp. 1221–1227 (2011)Google Scholar
  32. 32.
    Holz, D., Behnke, S.: Fast range image segmentation and smoothing using approximate surface reconstruction and region growing. In: Proceedings of the 12th International Conference on Intelligent Autonomous Systems (IAS), Jeju Island (2012)Google Scholar
  33. 33.
    Foley, J.D., van Dam, A., Feiner, S.K., Hughes, J.F.: Computer graphics: principles and practice, 2nd edn. Addison-Wesley Longman Publishing Co., Inc., Boston (1990)Google Scholar
  34. 34.
    Wang, L., Huang, M., El-Ghazawi, T.A.: Exploiting concurrent kernel execution on graphic processing units. In: Smari, W.W., McIntire, J.P. (eds.) HPCS, IEEE, pp. 24–32 (2011)Google Scholar
  35. 35.
    NVIDIA. Visual profiler (2012)Google Scholar
  36. 36.
    Harris, M.: Optimizing parallel reduction in CUDA. NVIDIA Dev. Technology (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Sergio Orts-Escolano
    • 1
    Email author
  • Vicente Morell
    • 1
  • Jose Garcia-Rodriguez
    • 1
  • Miguel Cazorla
    • 1
  • Robert B. Fisher
    • 2
  1. 1.Computer Technology DepartmentUniversity of AlicanteAlicanteSpain
  2. 2.Institute of Perception, Action and BehaviourUniversity of EdinburghEdinburghUK

Personalised recommendations