Learning to Fuse 3D+2D Based Face Recognition at Both Feature and Decision Levels

  • Stan Z. Li
  • ChunShui Zhao
  • Meng Ao
  • Zhen Lei
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3723)


2D intensity images and 3D shape models are both useful for face recognition, but in different ways. While algorithms have long been developed using 2D or 3D data, recently has seen work on combining both into multi-modal face biometrics to achieve higher performance. However, the fusion of the two modalities has mostly been at the decision level, based on scores obtained from independent 2D and 3D matchers.

In this paper, we propose a systematic framework for fusing 2D and 3D face recognition at both feature and decision levels, by exploring synergies of the two modalities at these levels. The novelties are the following. First, we propose to use Local Binary Pattern (LBP) features to represent 3D faces and present a statistical learning procedure for feature selection and classifier learning. This leads to a matching engine for 3D face recognition. Second, we propose a statistical learning approach for fusing 2D and 3D based face recognition at both feature and decision levels. Experiments show that the fusion at both levels yields significantly better performance than fusion at the decision level.


Face Recognition Face Image Local Binary Pattern Decision Level Principal Component Analysis Method 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Turk, M.A., Pentland, A.P.: Eigenfaces for recognition. Journal of Cognitive Neuroscience 3, 71–86 (1991)CrossRefGoogle Scholar
  2. 2.
    Belhumeur, P.N., Hespanha, J.P., Kriegman, D.J.: Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence 19, 711–720 (1997)CrossRefGoogle Scholar
  3. 3.
    Bartlett, M.S., Lades, H.M., Sejnowski, T.J.: Independent component representations for face recognition. In: Proceedings of the SPIE, Conference on Human Vision and Electronic Imaging III, vol. 3299, pp. 528–539 (1998)Google Scholar
  4. 4.
    Lades, M., Vorbruggen, J., Buhmann, J., Lange, J., von der Malsburg, C., Wurtz, R.P., Konen, W.: Distortion invariant object recognition in the dynamic link architecture. IEEE Transactions on Computers 42, 300–311 (1993)CrossRefGoogle Scholar
  5. 5.
    Wiskott, L., Fellous, J., Kruger, N., van der Malsburg, C.: Face recognition by elastic bunch graph matching. IEEE Transactions on Pattern Analysis and Machine Intelligence 19, 775–779 (1997)CrossRefGoogle Scholar
  6. 6.
    Ojala, T., Pietikainen, M., Harwood, D.: A comparative study of texture measures with classification based on feature distributions. Pattern Recognition 29, 51–59 (1996)CrossRefGoogle Scholar
  7. 7.
    Ahonen, T., Hadid, A., Pietikainen, M.: Face recognition with local binary patterns. In: Proceedings of the European Conference on Computer Vision, Prague, Czech, pp. 469–481 (2004)Google Scholar
  8. 8.
    Hadid, A., Pietikinen, M., Ahonen, T.: A discriminative feature space for detecting and recognizing faces. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 797–804 (2004)Google Scholar
  9. 9.
    Jones, M., Viola, P.: Face recognition using boosted local features. Tech. Report TR2003-025, MERL (2003)Google Scholar
  10. 10.
    Zhang, G., Huang, X., Li, S.Z., Wang, Y.: Boosting local binary pattern (LBP)-based face recognition. In: Li, S.Z., Lai, J.-H., Tan, T., Feng, G.-C., Wang, Y. (eds.) SINOBIOMETRICS 2004. LNCS, vol. 3338, pp. 180–187. Springer, Heidelberg (2004)Google Scholar
  11. 11.
    Kanade, T.: Picture Processing by Computer Complex and Recognition of Human Faces. PhD thesis, Kyoto University (1973)Google Scholar
  12. 12.
    Cartoux, J.Y., LaPreste, J.T., Richetin, M.: Face authentication or recognition by profile extraction from range images. In: Proceedings of the Workshop on Interpretation of 3D Scenes (1989)Google Scholar
  13. 13.
    Besl, P.J., Jain, R.C.: Intrinsic and extrinsic surface characteristics. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, California, pp. 226–233 (1985)Google Scholar
  14. 14.
    Bowyer, K.W., Chang, K., Flynn, P.: A survey of 3D and multi-modal 3d+2d face recognition. In: Proceedings of International Conference Pattern Recognition, pp. 358–361 (2004)Google Scholar
  15. 15.
    Lu, X., Jain, A.K.: Integrating range and texture information for 3d face recognition. In: Proc. 7th IEEE Workshop on Applications of Computer Vision (WACV 2005), Breckenridge, CO. (2005)Google Scholar
  16. 16.
    Tsalakanidou, F., Malassiotis, S., Strintzis, M.G.: Face localization and authentication using color and depth images 14, 152–168 (2005)Google Scholar
  17. 17.
    Chang, K.I., Bowyer, K.W., Flynn, P.J.: An evaluation of multi-modal 2D+3D face biometrics. IEEE Transactions on Pattern Analysis and Machine Intelligence (2005) (to appear)Google Scholar
  18. 18.
    Freund, Y., Schapire, R.: A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55, 119–139 (1997)zbMATHCrossRefMathSciNetGoogle Scholar
  19. 19.
    Schapire, R.: A brief introduction to boosting. In: Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence (1999)Google Scholar
  20. 20.
    Viola, P., Jones, M.: Robust real time object detection. In: IEEE ICCV Workshop on Statistical and Computational Theories of Vision, Vancouver, Canada (2001)Google Scholar
  21. 21.
    Moghaddam, B., Nastar, C., Pentland, A.: A Bayesain similarity measure for direct image matching. Media Lab Tech. Report No.393, MIT (1996)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Stan Z. Li
    • 1
  • ChunShui Zhao
    • 1
  • Meng Ao
    • 1
  • Zhen Lei
    • 1
  1. 1.Center for Biometrics and Security Research & National Laboratory of Pattern Recognition, Institute of AutomationChinese Academy of SciencesBeijingChina

Personalised recommendations