Facial Occlusion Reconstruction: Recovering Both the Global Structure and the Local Detailed Texture Components

  • Ching-Ting Tu
  • Jenn-Jier James Lien
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4872)


An automatic facial occlusion reconstruction system based upon a novel learning algorithm called the direct combined model (DCM) approach is presented. The system comprises two basic DCM modules, namely a shape reconstruction module and a texture reconstruction module. Each module models the occluded and non-occluded regions of the facial image in a single, combined eigenspace, thus preserving the correlations between the geometry of the facial features and the pixel grayvalues, respectively, in the two regions. As a result, when shape or texture information is available only for the non-occluded region of the facial image, the optimal shape and texture of the occluded region can be reconstructed via a process of Bayesian inference within the respective eigenspaces. To enhance the quality of the reconstructed results, the shape reconstruction module is rendered robust to facial feature point labeling errors by suppressing the effects of biased noises. Furthermore, the texture reconstruction module recovers the texture of the occluded facial image by synthesizing the global texture image and the local detailed texture image. The experimental results demonstrate that compared to existing facial reconstruction systems, the reconstruction results obtained using the proposed DCM-based scheme are quantitatively closer to the ground truth.


Facial reconstruction facial synthesis eigenspace facial occlusion 


  1. 1.
    Blanz, V., Romdhani, S., Vetter, T.: Face Identification across Different Poses and Illuminations with a 3D Morphable Model. In: IEEE Intl. Conf. on FG, pp. 202–207 (2002)Google Scholar
  2. 2.
    Chen, H., Xu, Y.Q., Shum, H.Y., Zhu, S.C., Zhen, N.N.: Example-Based Facial Sketch Generation with Non-Parametric Sampling. In: Proceedings of ICCV, pp. 433–438 (2001)Google Scholar
  3. 3.
    Cootes, T.F., Taylor, C.J.: Statistical Models of Appearance for Computer Vision. Technical Report, Univ. of Manchester (2000) Google Scholar
  4. 4.
    Fisker, R.: Making Deformable Template Models Operational. PhD Thesis, Informatics and Mathematical Modelling, Technical University of Denmark (2000) Google Scholar
  5. 5.
    Freeman, W.T., Pasztor, E.C.: Learning Low-Level Vision. In: ICCV, pp. 1182–1189 (1999)Google Scholar
  6. 6.
    Golub, G.H., Van Loan, C.F.: Matrix Computations, 3rd edn. Johns Hopkins University Press, Baltimore (1996)zbMATHGoogle Scholar
  7. 7.
    Hwang, B.W., Blanz, V., Vetter, T., Lee, S.W.: Face Reconstruction from a Small Number of Feature Points. In: Proceedings of ICPR, pp. 842–845 (2000)Google Scholar
  8. 8.
    Hwang, B.W., Lee, S.W.: Reconstruction of Partially Damaged Face Images Based on a Morphable Face Model. IEEE Trans. on PAMI 25(3), 365–372 (2003)Google Scholar
  9. 9.
    Jones, M.J., Poggio, T.: Multidimensional Morphable Models: A Framework for Representing and Matching Object Classes. IJCV 29(2), 107–131 (1998)CrossRefGoogle Scholar
  10. 10.
    Lanitis, A.: Person Identification from Heavily Occluded Face Images. In: Handschuh, H., Hasan, M.A. (eds.) SAC 2004. LNCS, vol. 3357, pp. 5–9. Springer, Heidelberg (2004)Google Scholar
  11. 11.
    Liu, C., Shum, H.Y., Zhang, C.S.: A Two-Step Approach to Hallucinating Faces: Global Parametric Model and Local Nonparametric Model. In: Proc. of CVPR, pp. 192–198 (2001)Google Scholar
  12. 12.
    Mo, Z., Lewis, J.P., Neumann, U.: Face Inpainting with Local Linear Representations. In: Proceedings of British Machine Vision Conference, vol. 1, pp. 347–356 (2004)Google Scholar
  13. 13.
    Park, J.S., Oh, Y.H., Ahn, S.C., Lee, S.W.: Glasses Removal from Facial Image Using Recursive PCA Reconstruction. IEEE Trans. on PAMI 27(5), 805–811 (2005)Google Scholar
  14. 14.
    Saito, Y., Kenmochi, Y., Kotani, K.: Estimation of Eyeglassless Facial Images Using Principal Component Analysis. In: Proceedings of ICIP, vol. 4, pp. 197–201 (1999)Google Scholar
  15. 15.
    Vetter, T., Poggio, T.: Linear Object Classes and Image Synthesis from a Single Example Image. IEEE Trans. on PAMI 19(7), 733–742 (1997)Google Scholar
  16. 16.
    Wu, C., Liu, C., Shum, H.Y., Xu, Y.Q., Zhang, Z.: Automatic Eyeglasses Removal from Face Images. IEEE Trans. PAMI 26, 322–336 (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Ching-Ting Tu
    • 1
  • Jenn-Jier James Lien
    • 1
  1. 1.Robotics Laboratory, Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan, TaiwanR.O.C.

Personalised recommendations