Advertisement

Springer Nature is making Coronavirus research free. View research | View latest news | Sign up for updates

From 3D Point Clouds to Pose-Normalised Depth Maps

Abstract

We consider the problem of generating either pairwise-aligned or pose-normalised depth maps from noisy 3D point clouds in a relatively unrestricted poses. Our system is deployed in a 3D face alignment application and consists of the following four stages: (i) data filtering, (ii) nose tip identification and sub-vertex localisation, (iii) computation of the (relative) face orientation, (iv) generation of either a pose aligned or a pose normalised depth map. We generate an implicit radial basis function (RBF) model of the facial surface and this is employed within all four stages of the process. For example, in stage (ii), construction of novel invariant features is based on sampling this RBF over a set of concentric spheres to give a spherically-sampled RBF (SSR) shape histogram. In stage (iii), a second novel descriptor, called an isoradius contour curvature signal, is defined, which allows rotational alignment to be determined using a simple process of 1D correlation. We test our system on both the University of York (UoY) 3D face dataset and the Face Recognition Grand Challenge (FRGC) 3D data. For the more challenging UoY data, our SSR descriptors significantly outperform three variants of spin images, successfully identifying nose vertices at a rate of 99.6%. Nose localisation performance on the higher quality FRGC data, which has only small pose variations, is 99.9%. Our best system successfully normalises the pose of 3D faces at rates of 99.1% (UoY data) and 99.6% (FRGC data).

This is a preview of subscription content, log in to check access.

References

  1. Ankerst, M., Kastenmüller, G., Kriegel, H.-P., & Seidl, T. (1999). 3d shape histograms for similarity search and classification in spatial databases. In 6th international symposium in advances in spatial databases (pp. 207–226).

  2. Arun, K. S., Huang, T. S., & Blostein, S. D. (1987). Least-squares fitting of two 3d point sets. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9(5), 698–700.

  3. Assfalg, J., Bimbo, A. D., & Pala, P. (2004). Spin images for retrieval of 3d objects by local and global similarity. In Proc. 17th int. conf. on pattern recognition (ICPR’04) (vol. 3, pp. 906–909).

  4. Belhumeur, P. N., Hespanha, J., & Kriegman, D. J. (1997). Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7), 711–720.

  5. Besl, P., & Jain, R. C. (1985). Three-dimensional object recognition. ACM Computing Surveys, 17(1), 75–145.

  6. Besl, P., & McKay, N. D. (1992). A method for registration of 3D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2), 239–256.

  7. Blanz, V., & Vetter, T. (2003). Face recognition based on fitting a 3d morphable model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(9), 1063–1074.

  8. Bowyer, K. W., Chang, K. I., & Flynn, P. J. (2006). A survey of approaches and challenges in 3d and multi-modal 3d+2d face recognition. Computer Vision and Image Understanding, 101(1), 1–15.

  9. Bronstein, A. M., Bronstein, M. M., & Kimmel, R. (2007). Expression-invariant representation of faces. IEEE Transactions on Image Processing, 16(1), 188–197.

  10. Carr, J., Fright, W.R., & Beatson, R. K. (1997). Surface interpolation with radial basis functions for medical imaging. IEEE Transactions on Medical Imaging, 16(1), 96–107.

  11. Carr, J. C., Beatson, R. K., Cherrie, J. B., Mitchell, T. J., Fright, W. R., McCallum, B. C., & Evans, T. (2001). Reconstruction and representation of 3d objects with radial basis functions. In Proc. of ACM Siggraph 2001 (pp. 67–76).

  12. Chang, K. I., Bowyer, K. W., & Flynn, P. J. (2005). An evaluation of multimodal 2d+3d face biometrics. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(4), 619–624.

  13. Chang, K. I., Bowyer, K. W., & Flynn, P. J. (2006). Multiple nose region matching for 3d face recognition under varying facial expression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(10), 1695–1700.

  14. Chen, C., & Prakash, E. (2005). Face personalization: animated face modeling approach using radial basis function. In TENCON 2005 IEEE Region 10 (pp. 1–6) November 2005.

  15. Chen, D.-Y., Tian, X.-P., Shen, Y.-T., & Ouhyoung, M. (2003). On visual similarity based 3d model retrieval. Eurographics, 22(3), 2003.

  16. Chetverikov, D., Stepanov, D., & Krsek, P. (2005). Robust Euclidean alignment of 3d point sets: the trimmed iterative closest point algorithm. Image and Vision Computing, 23(3), 299–309.

  17. Chin-Seng Chua, F. H., & Ho, Y.-K. (2001). 3d human face recognition using point signature. In 4th IEEE int. conf. on automatic face and gesture recognition 2000 (pp. 233–238).

  18. Chua, C. S., & Jarvis, R. (1997). Point signatures: A new representation for 3D object recognition. International Journal of Computer Vision, 25(1), 63–85.

  19. Colbry, D., Stockman, D., & Jain, A. (2005). Detection of anchor points for 3d face verification. In IEEE Computer Society conference on computer vision and pattern recognition. Workshops (p 118).

  20. Conde, C., Cipolla, R., Rodriguez-Aragon, L. J., Serrano, A., & Cabello, E. (2005). 3d facial feature location with spin images. In IAPR conf. on machine vision applications (MVA’05) (pp. 418–421).

  21. Dinh, H. Q., & Kropac, S. (2006). Multi-resolution spin-images. In Proc. IEEE conf. on computer vision and pattern recognition (CVPR’06) (pp. 863–870).

  22. Dorai, C., & Jain, A. K. (1997). Cosmos-a representation scheme for 3d free-form objects. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(10), 1115–1130.

  23. Faugeras, O. D., & Hebert, M. (1986). The representation, recognition and locating of 3d objects. International Journal of Robotics Research, 5(3), 27–52.

  24. Franke, R. (1982). Scattered data interpolation: tests of some methods. Mathematics of Computation, 38(157), 181–200.

  25. Funkhouser, T., Min, P., Kazhdan, M., Chen, J., Halderman, A., Dobkin, D., & Jacobs, D. (2003). A search engine for 3d models. ACM Transactions on Graphics, 22, 83–105.

  26. Gordon, G. G. (1992). Face recognition based on depth and curvature features. In Proc. IEEE computer society conf. on computer vision and pattern recognition (pp. 808–810).

  27. Greengard, L., & Rokhlin, V. (1987). A fast algorithm for particle simulations. Journal of Computer Physics, 73, 325–348.

  28. Haralick, R. M., Joo, H., Lee, C.-N., Zhuang, X., Vaidya, V. G., & Kim, M. B. (1989). Pose estimation from corresponding point data. IEEE Transactions on Systems, Man and Cybernetics, 19(6), 1426–1446.

  29. Heseltine, T., Pears, N. E., & Austin, J. (2004a). Three-dimensional face recognition: a fishersurface approach. In Lecture notes in computer science: Vol. 3212. Proc. of int. conf. on image analysis and recognition, part II (pp. 684–691). Berlin: Springer.

  30. Heseltine, T., Pears, N. E., & Austin, J. (2004b). Three-dimensional face recognition: an eigensurface approach. In Proc. IEEE int. conf. on image processing (pp. 1–2).

  31. Heseltine, T., Pears, N. E., & Austin, J. (2008). Three-dimensional face recognition using combinations of surface feature map subspace components. Image and Vision Computing, 26(3), 382–396.

  32. Horn, B. K. P. (1984). Extended Gaussian images. Proceedings of the IEEE, 72(2), 1671–1686.

  33. Hou, Q., & Bai, L. (2005). Line feature detection from 3d point clouds via adaptive cs-rbfs shape reconstruction and multistep vertex normal manipulation. In: International conference on computer graphics, imaging and vision: new trends (pp. 79–83), July 2005.

  34. Hu, X., Tang, Y., & Zhang, Z. (2008). Video object matching based on sift algorithm. In IEEE int. conference on neural networks and signal processing (pp. 412–415), June 2008.

  35. Johnson, A. E., & Hebert, M. (1997). Using spin images for efficient object recognition in cluttered 3d scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(5), 433–449.

  36. Kakadiaris, I., Passalis, G., Toderici, G., Murtuza, N., & Theoharis, T. (2006). 3d face recognition. In British machine vision conference (BMVC’06).

  37. Kazhdan, M. M., Funkhouser, T. A., & Rusinkiewicz, S. (2003). Rotation invariant spherical harmonic representation of 3d shape descriptors. In Symposium on geometry processing (pp. 156–165).

  38. Kimmel, R., Bronstein, A. M., & Bronstein, M. M. (2005). Three-dimensional face recognition. International Journal of Computer Vision, 64(1), 5–30.

  39. Lorensen, W. E., & Cline, H. E. (1987). Marching cubes: a high resolution 3d surface construction algorithm. Computer Graphics, 21(4), 163–169.

  40. Lowe, D. G. (1999). Object recognition from local scale-invariant features. In 7th IEEE int. conf. on computer vision (Vol. 2, pp. 1150–1157), September 1999.

  41. Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60, 91–110.

  42. Lu, X., Jain, A. K., & Colbry, D. (2006). Matching 2.5d face scans to 3d models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(1), 31–43.

  43. Mian, A. S., Bennamoun, M., & Owens, R. (2007). An efficient multimodal 2d–3d hybrid approach to automatic face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(11), 1927–1943.

  44. Mian, A. S., Bennamoun, M., & Owens, R. (2008). Keypoint detection and local feature matching for textured 3d face recognition. International Journal of Computer Vision, 79(1), 1–12.

  45. Papadakis, P., Pratikakis, I., Perantonis, S., & Theoharis, T. (2007). Efficient 3d shape matching and retrieval using a concrete radialized spherical projection representation. Pattern Recognition, 40(9), 2437–2452.

  46. Pears, N. E. (2008). Rbf shape histograms and their application to 3d face processing. In 8th IEEE int. conf. on automatic face and gesture recognition (FG’08), Amsterdam, Netherlands.

  47. Pears, N. E., & Heseltine, T. D. (2006). Isoradius contours: New representations and techniques for 3d face matching and registration. In 3rd int. symposium on 3D data processing, visualization and transmission (3DPVT’06), University of North Carolina, USA (pp. 176–183).

  48. Phillips, P. J., Flynn, P. J., Scruggs, T., Bowyer, K. W., Chang, J., Hoffman, K., Marques, J., Min, J., & Worek, W. (2005). Overview of the face recognition grand challenge. In IEEE conf. on computer vision and pattern recognition (pp. 947–954).

  49. Rohling, R., Gee, A., Berman, L., & Treece, G. (1999). Radial basis function interpolation for freehand 3d ultrasound. In Lecture Notes in Computer Science : Vol. 1613. Information processing in medical imaging (pp. 478–483). Berlin: Springer.

  50. Saupe, D., & Vranic, D. V. (2001). 3d model retrieval with spherical harmonics and moments. In Proceedings of the DAGM symposium on pattern recognition (pp. 392–397). Berlin: Springer.

  51. Savchenko, V. V., Pasko, A., Okunev, O. G., & Kunii, T. L. (1985). Function representation of solids reconstructed from scattered surface points and contours. Computer Graphics Forum, 14(4), 181–188.

  52. Se, S., Lowe, D. G., & Little, J. (2002). Mobile robot localization mapping with uncertainty using scale-invariant visual landmarks. International Journal of Robotics Research, 21(8), 735–758.

  53. Segundo, M., Queirolo, C., Bellon, O., & Silva, L. (2007). Automatic 3d facial segmentation and landmark detection. In Proc. of 14th int. conf. on image analysis and processing (pp. 431–436).

  54. Shilane, P., Min, P., Kazhdan, M., & Funkhouser, T. (2004). The Princeton shape benchmark. In Shape modeling and applications (pp. 167–178).

  55. Stein, F., & Medioni, G. (1992). Structural indexing: efficient 3-d object recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2), 125–145.

  56. Theoharis, T. (2008). 3d object retrieval, inter-class vs. intra-class. In Artificial intelligence techniques for computer graphics (pp. 55–66). Berlin: Springer.

  57. Theoharis, T., Passalis, G., Toderici, G., & Kakadiaris, I. A. (2008). Unified 3d face and ear recognition using wavelets on geometry images. Pattern Recognition, 41(3), 796–804.

  58. Turk, G., & O’Brien, J. (1999). Shape transformation using variational implicit surfaces. In Proc. ACM on computer graphics SIGGRAPH 1999 (pp. 335–342).

  59. Turk, M., & Pentland, A. (1991). Eigenfaces for recognition. Journal of Cognitive Neuroscience, 3(1), 71–86.

  60. Wang, Y., Chua, C., & Ho, Y. (2002). Facial feature detection and face recognition from 2d and 3d images. Pattern Recognition Letters, 23(10), 1191–1202.

  61. Whitmarsh, T., Veltkamp, R. C., Spagnuolo, M., Marini, S., & Harr, F. T. (2006). Landmark detection on 3d face scans by facial model registration. In 1st international symposium on shapes and semantics (pp. 71–75).

  62. Xu, C., Tan, T., Wang, Y., & Quan, L. (2006). Combining local features for robust nose location in 3d facial data. Pattern Recognition Letters, 27, 1487–1494.

Download references

Author information

Correspondence to Nick Pears.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Pears, N., Heseltine, T. & Romero, M. From 3D Point Clouds to Pose-Normalised Depth Maps. Int J Comput Vis 89, 152–176 (2010). https://doi.org/10.1007/s11263-009-0297-y

Download citation

Keywords

  • 3D feature extraction
  • Invariance
  • 3D landmark localisation
  • 3D pose normalisation