3D Research

, 2:6 | Cite as

Multi-sensor 3D volumetric reconstruction using CUDA

3DR Express

Abstract

This paper presents a full-body volumetric reconstruction of a person in a scene using a sensor network, where some of them can be mobile. The sensor network is comprised of couples of camera and inertial sensor (IS). Taking advantage of IS, the 3D reconstruction is performed using no planar ground assumption. Moreover, IS in each couple is used to define a virtual camera whose image plane is horizontal and aligned with the earth cardinal directions. The IS is furthermore used to define a set of inertial planes in the scene. The image plane of each virtual camera is projected onto this set of parallel-horizontal inertial-planes, using some adapted homography functions. A parallel processing architecture is proposed in order to perform human real-time volumetric reconstruction. The real-time characteristic is obtained by implementing the reconstruction algorithm on a graphics processing unit (GPU) using Compute Unified Device Architecture (CUDA). In order to show the effectiveness of the proposed algorithm, a variety of the gestures of a person acting in the scene is reconstructed and demonstrated. Some analyses have been carried out to measure the performance of the algorithm in terms of processing time. The proposed framework has potential to be used by different applications such as smart-room, human behavior analysis and 3D teleconference.

Keywords

multi-view camera auto-stereoscopic visualization dynamic scenes 3D rendering quality assessment visual servoing 

References

  1. 1.
    H. Aliakbarpour J. Dias (2010) IMU-aided 3D Reconstruction based on Multiple Virtual Planes, DICTA'10 (the Australian Pattern Recognition and Computer Vision Society Conference), IEEE Pr, 1–3 December, Sydney, Australia.Google Scholar
  2. 2.
    S. M. Khan, P. Yan, M. Shah (2007) A Homographic Framework for the Fusion of Multi-view Silhouettes, Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference.Google Scholar
  3. 3.
    B. Michoud, E. Guillou, S. Bouakaz (2007) Real-Time and Markerless 3D Human Motion Capture Using Multiple Views, Human Motion-Understanding, Modeling, Capture and Animation, Springer Berlin/Heidelberg, 4814/2007:88–103CrossRefGoogle Scholar
  4. 4.
    Q. Zhang, H. Wang, S. Wei (2003) A NEW ALGORITHM FOR 3D PROJECTIVE RECONSTRUCTION BASED ON INFINITE HOMOGRAPHY. Machine Learning and Cybernetics, 2003 International Conference on, IEEE.Google Scholar
  5. 5.
    Z. Zhang, A. R. Hanson (1996) 3D Reconstruction Based on Homography Mapping, In ARPA Image Understanding Workshop.Google Scholar
  6. 6.
    M. Sormann, C. Zach, J. Bauer, K. Karner, H. Bishof (2007) Watertight Multi-view Reconstruction Based on Volumetric Graph-Cuts. In Ersball, Bjarne and Pedersen, Kim, editors, Image Analysis in Lecture Notes in Computer Science, Springer Berlin, Heidelberg, 393–402.Google Scholar
  7. 7.
    P. Lai, A. Yilmaz (2008) PROJECTIVE RECONSTRUCTION OF BUILDING SHAPE FROM SILHOUETTE IMAGES ACQUIRED FROM UNCALIBRATED CAMERAS. ISPRS Congress Beijing 2008, Proceedings of Commission III.Google Scholar
  8. 8.
    T. Feldmann, I. Mihailidis, S. Schulz, D. Paulus, A. Worner (2010) Online Full Body Human Motion Tracking Based on Dense Volumetric 3DÂ Reconstructions from Multi Camera Setups. In Dillmann, Rudiger and Beyerer, Jurgen and Hanebeck, Uwe and Schultz, Tanja, editors, KI 2010, Advances in Artificial Intelligence in Lecture Notes in Computer Science, Springer Berlin/Heidelberg, 74–81.CrossRefGoogle Scholar
  9. 9.
    R. Guerchouche, O. Bernier, T. Zaharia (2008) Multiresolution volumetric 3D object reconstruction for collaborative interactions. Pattern Recognition and Image Analysis, 18:621–637. 10.1134/S1054661808040147.CrossRefGoogle Scholar
  10. 10.
    T. Azevedo, J. Tavares, M. Vaz (2009) 3D Object Reconstruction from Uncalibrated Images Using an Off-the-Shelf Camera. Advances in Computational Vision and Medical Image Processing; in series of Computational Methods in Applied Sciences, Springer Netherlands, 13:117–136, Universidade do Porto.Google Scholar
  11. 11.
    H. Lee, A. Yilmaz (2010) 3D RECONSTRUCTION USING PHOTO CONSISTENCY FROM UNCALIBRATED MULTIPLE VIEWS. VISAPP 2010 — The International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications Google Scholar
  12. 12.
    B. Zhang, Y.F. Li (2005) An efficient method for dynamic calibration and 3D reconstruction using homographic transformation. Sensors and Actuators A: Physical, 119(2):349–357CrossRefGoogle Scholar
  13. 13.
    H. Lin, J. Wu (2008) 3D Reconstruction by Combining Shape from Silhouette with Stereo. IEEE.Google Scholar
  14. 14.
    B. Michoud, S. Bouakaz, E. Guillou. H. Briceno (2008) Largest Silhouette-Equivalent Volume for 3D Shapes Modeling without Ghost Object. M2SFA2 2008: Workshop on Multi-camera and Multi-modal Sensor Fusion, Marseille, France.Google Scholar
  15. 15.
    H. Aliakbarpour, J. Dias. (2010) Human Silhouette Volume Reconstruction Using a Gravity-based Virtual Camera Network. Proceedings of the 13th International Conference on Information Fusion, 26–29 July 2010 EICC Edinburgh, UK.Google Scholar
  16. 16.
    A. Calbi, C. Regazzoni, L. Marcenaro (2006) Dynamic Scene Reconstruction for Efficient Remote Surveillance. IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS'06).Google Scholar
  17. 17.
    J. Franco, E. Boyer (2005) Fusion of Multi-View Silhouette Cues Using a Space Occupancy Grid. Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV05).Google Scholar
  18. 18.
    J. F. Ferreira, J. Lobo, J. Dias (2010) Bayesian real-time perception algorithms on GPU — Real-time implementation of Bayesian models for multimodal perception using CUDA. Journal of Real-Time Image Processing, Special Issue.Google Scholar
  19. 19.
    L. Almeida, P. Menezes, J. Dias (2011) Stereo Vision Head Vergence Using GPU Cepstral Filtering. Proceedings of the Fifth International Conference on Computer Vision Theory and Applications (VISAPP), Vilamoura, Algarve, Portugal, March, 5–7.Google Scholar
  20. 20.
    A. Griesser, S. D. Roeck, A. Neubeck, L. V. Gool (2005) Gpu-based foreground-background segmentation using an extended colinearity criterion. In Proc. Vision, Modeling, and Visualization (VMV) 2005. Amsterdam, The Netherlands: IOS, Nov. 2005., 319–326Google Scholar
  21. 21.
    G. Ziegler (2010) GPU Data Structures for Graphics and Vision. PhD thesis, Max-Planck-Institut für Informatik.Google Scholar
  22. 22.
    C. Nitschke, A. Nakazawa, H. Takemura (2007) Real-Time Space Carving Using Graphics Hardware. IEICE — Trans. Inf. Syst., E90-D:1175–1184CrossRefGoogle Scholar
  23. 23.
    S.S. Stone, J.P. Haldar, S.C. Tsao, W.-m.W. Hwu, B.P. Sutton, Z.-P. Liang (2008) Accelerating advanced MRI reconstructions on GPUs. Journal of Parallel and Distributed Computing, 68(10):1307–1318, General-Purpose Processing using Graphics Processing Units.CrossRefGoogle Scholar
  24. 24.
    W. Waizenegger, I. Feldmann, P. Eisert, P. Kauff (2009) Parallel high resolution real-time Visual Hull on GPU. Image Processing (ICIP), 2009 16th IEEE International Conference on, 430–4304Google Scholar
  25. 25.
    S. Yous, H. Laga, M. Kidode, K. Chihara (2007) GPU-based shape from silhouettes. Proceedings of the 5th international conference on Computer graphics and interactive techniques in Australia and Southeast Asia in GRAPHITE’ 07, 71–77, New York, NY, USA, 2007. ACM.Google Scholar
  26. 26.
    D. Knoblauch, F. Kuester (2009) Focused Volumetric Visual Hull with Color Extraction. In Bebis, George and Boyle, Richard and Parvin, Bahram and Koracin, Darko and Kuno, Yoshinori and Wang, Junxian and Pajarola, Renato and Lindstrom, Peter and Hinkenjann, Andre and Encarnacao, Miguel and Silva, Claudio and Coming, Daniel, editors, Advances in Visual Computing in Lecture Notes in Computer Science, Springer Berlin-Heidelberg, 208–217Google Scholar
  27. 27.
    A. Ladikos, S. Benhimane, N. Navab (2008) Efficient visual hull computation for real-time 3D reconstruction using CUDA. Computer Vision and Pattern Recognition Workshops, 2008. CVPRW’ 08. IEEE Computer Society Conference on, 1–8.Google Scholar
  28. 28.
    M. Yguel, O. Aycard, C. Laugier (2006) Efficient GPU-based Construction of Occupancy Grids Using several Laser Range-finders. Oct. 2006. Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on.Google Scholar
  29. 29.
    M. Kalantari, A. Hashemi, F. Jung, J. Guedon (2011) A New Solution to the Relative Orientation Problem Using Only 3 Points and the Vertical Direction. Journal of Mathematical Imaging and Vision, 39:259–268CrossRefMathSciNetGoogle Scholar
  30. 30.
    H. Aliakbarpour, J. Dias (2011) Inertial-Visual Fusion For Camera Network Calibration. IEEE 9th International Conference on Industrial Informatics (INDIN 2011).Google Scholar
  31. 31.
    J. Lobo, J. Dias (2003) Vision and inertial sensor cooperation using gravity as a vertical reference. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 25(12):1597–1608CrossRefGoogle Scholar
  32. 32.
    M. Labrie, P. Hebert, (2007) Efficient camera motion and 3D recovery using an inertial sensor. Computer and Robot Vision, 2007. CRV’ 07. Fourth Canadian Conference on, 55–62Google Scholar
  33. 33.
    T. Okatani, K. Deguchi (2002) Robust estimation of camera translation between two images using a camera with a 3D orientation sensor. Pattern Recognition, 2002. Proceedings. 16th International Conference on, 1:275–278Google Scholar
  34. 34.
    M. A. Brodie, A. Walmsley, W. Page (2008) The static accuracy and calibration of inertial measurement units for 3D orientation. Computer Methods in Biomechanics and Biomedical Engineering, 11:641–648CrossRefGoogle Scholar
  35. 35.
    R. Hartley, A. Zisserman (2003) Multiple View Geometry in Computer Vision. CAMBRIDGE UNIVERSITY PRESS Google Scholar
  36. 36.
    Y. Ma, S. Soatta, J. Kosecka, S. S. Sastry (2004) An invitation to 3D vision. Springer.Google Scholar
  37. 37.
    L. G. B. Mirisola, J. Dias, A. Traca de Almeida (2007) Trajectory Recovery and 3D Mapping from Rotation-Compensated Imagery for an Airship. Proceedings of the 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems San Diego, CA, USA, Oct 29–Nov 2.Google Scholar
  38. 38.
    L. G. B. Mirisola (2009) Exploiting attitude sensing in vision-based navigation, mapping and tracking including results from an airship. PhD thesis.Google Scholar
  39. 39.
    L. G. B. Mirisola, J. M. M. Dias (2007) Exploiting inertial sensing in mosaicing and visual navigation. In 6th IFAC Symposium on Inteligent Autonomous Vehicles (IAV07), Toulouse, France, Sep. 2007.Google Scholar
  40. 40.
  41. 41.
  42. 42.
    Xsens Motion Technologies. http://www.xsens.com.
  43. 43.
    J. Bouguet (2003) Camera Calibration Toolbox for Matlab. www.vision.caltech.edu/bouguetj.Google Scholar
  44. 44.
    J. Lobo, J. Dias (2007) Relative Pose Calibration Between Visual and Inertial Sensors. International Journal of Robotics Research, Special Issue 2nd Workshop on Integration of Vision and Inertial Sensors, 26:561–575Google Scholar
  45. 45.
    P. Kakumanu, S. Makrogiannis, N. Bourbakis (2007) A survey of skin-color modeling and detection methods. Pattern Recogn, 40:1106–1122CrossRefMATHGoogle Scholar
  46. 46.
    G. R. Bradski (1998) Computer Vision Face Tracking For Use in a Perceptual User Interface. Intel Technology Journal, (Q2).Google Scholar
  47. 47.
  48. 48.
  49. 49.
    T. Wada, X. Wu, S. Tokaim. T. Matsuyama (2000) Homography Based Parallel Volume Intersection: Toward Real-Time Volume Reconstruction Using Active Cameras. Computer Architectures for Machine Perception, 2000. Proceedings. Fifth IEEE International Workshop on 11–13 Sept. 2000, 331–339Google Scholar
  50. 50.
    P. Lai, A. Yilmaz (2008) Efficient object shape recovery via slicing planes. Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, 1–6Google Scholar

Copyright information

© 3D Display Research Center and Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  1. 1.Institute of Systems and Robotics, Polo IIUniversity of CoimbraCoimbraPortugal
  2. 2.Institute Polytechnic of TomarTomarPortugal

Personalised recommendations