Synergistic Face Detection and Pose Estimation with Energy-Based Models

  • Margarita Osadchy
  • Yann Le Cun
  • Matthew L. Miller
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4170)


We describe a novel method for real-time, simultaneous multi-view face detection and facial pose estimation. The method employs a convolutional network to map face images to points on a manifold, parametrized by pose, and non-face images to points far from that manifold. This network is trained by optimizing a loss function of three variables: image, pose, and face/non-face label. We test the resulting system, in a single configuration, on three standard data sets – one for frontal pose, one for rotated faces, and one for profiles – and find that its performance on each set is comparable to previous multi-view face detectors that can only handle one form of pose variation. We also show experimentally that the system’s accuracy on both face detection and pose estimation is improved by training for the two tasks together.


Face Detection Equal Error Rate Multitask Learning Synergy Test Pose Estimation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bottou, L., LeCun, Y.: The Lush Manual (2002),
  2. 2.
    Caruana, R.: Multitask learning. Machine Learning 28, 41–75 (1997)CrossRefGoogle Scholar
  3. 3.
    Garcia, C., Delakis, M.: A neural architecture for fast and robust face detection. In: IEEE-IAPR Int. Conference on Pattern Recognition, pp. 40–43 (2002)Google Scholar
  4. 4.
    Huang, F.J., LeCun, Y.: Loss functions for discriminative training of energy-based graphical models. Technical report, Courant Institute of Mathematical Science, NYU (June 2004)Google Scholar
  5. 5.
    Jones, M., Viola, P.: Fast multi-view face detection. Technical Report TR2003-96, Mitsubishi Electric Research Laboratories (2003)Google Scholar
  6. 6.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  7. 7.
    Li, S.Z., Zhu, L., Zhang, Z., Blake, A., Zhang, H., Shum, H.: Statistical learning of multi-view face detection. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2353, pp. 67–81. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  8. 8.
    Li, Y., Gong, S., Liddell, H.: Support vector regression and classification based multi-view face detection and recognition. In: Face and Gesture (2000)Google Scholar
  9. 9.
    Moon, H., Miller, M.L.: Estimating facial pose from sparse representation. In: International Conference on Image Processing, Singapore (2004)Google Scholar
  10. 10.
    Pentland, A., Moghaddam, B., Starner, T.: View-based and modular eigenspaces for face recognition. In: CVPR (1994)Google Scholar
  11. 11.
    Rowley, H.A., Baluja, S., Kanade, T.: Neural network-based face detection. PAMI 20, 22–38 (1998)Google Scholar
  12. 12.
    Rowley, H.A., Baluja, S., Kanade, T.: Rotation invariant neural network-based face detection. In: Computer Vision and Pattern Recognition (1998)Google Scholar
  13. 13.
    Schneidermn, H., Kanade, T.: A statistical method for 3d object detection applied to faces and cars. In: Computer Vision and Pattern Recognition (2000)Google Scholar
  14. 14.
    Sung, K., Poggio, T.: Example-based learning of view-based human face detection. PAMI 20, 39–51 (1998)Google Scholar
  15. 15.
    Vaillant, R., Monrocq, C., LeCun, Y.: Original approach for the localisation of objects in images. IEE Proc. on Vision, Image, and Signal Processing 141(4), 245–250 (1994)CrossRefGoogle Scholar
  16. 16.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings IEEE Conf. on Computer Vision and Pattern Recognition, pp. 511–518 (2001)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Margarita Osadchy
    • 1
  • Yann Le Cun
    • 2
  • Matthew L. Miller
    • 3
  1. 1.Computer Science DepartmentUniversity of HaifaHaifaIsrael
  2. 2.The Courant Institute of Mathematical SciencesNew York UniversityNew YorkUSA
  3. 3.NEC Labs AmericaPrincetonUSA

Personalised recommendations