International Journal of Computer Vision

, Volume 97, Issue 1, pp 91–103 | Cite as

Self-calibrated, Multi-spectral Photometric Stereo for 3D Face Capture

  • George Vogiatzis
  • Carlos Hernández


This paper addresses the problem of obtaining 3d detailed reconstructions of human faces in real-time and with inexpensive hardware. We present an algorithm based on a monocular multi-spectral photometric-stereo setup. This system is known to capture high-detailed deforming 3d surfaces at high frame rates and without having to use any expensive hardware or synchronized light stage. However, the main challenge of such a setup is the calibration stage, which depends on the lights setup and how they interact with the specific material being captured, in this case, human faces. For this purpose we develop a self-calibration technique where the person being captured is asked to perform a rigid motion in front of the camera, maintaining a neutral expression. Rigidity constrains are then used to compute the head’s motion with a structure-from-motion algorithm. Once the motion is obtained, a multi-view stereo algorithm reconstructs a coarse 3d model of the face. This coarse model is then used to estimate the lighting parameters with a stratified approach: In the first step we use a RANSAC search to identify purely diffuse points on the face and to simultaneously estimate this diffuse reflectance model. In the second step we apply non-linear optimization to fit a non-Lambertian reflectance model to the outliers of the previous step. The calibration procedure is validated with synthetic and real data.


Photometric stereo Multi-spectral Faces Motion capture Calibration 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Cyberware, inc. Accessed in March 2010 (2010)
  2. Dimensional Imaging. Accessed in March 2010 (2010)
  3. Mova. Accessed in March 2010 (2010)
  4. Basri, R., Jacobs, D., & Kemelmacher, I. (2007). Photometric stereo with general, unknown lighting. International Journal of Computer Vision, 72(3), 239–257. CrossRefGoogle Scholar
  5. Belhumeur, P. N., Kriegman, D. J., & Yuille, A. L. (1999). The bas-relief ambiguity. International Journal of Computer Vision, 35(1), 33–44. CrossRefGoogle Scholar
  6. Fischler, M., & Bolles, R. (1981). Random sample consensus: a paradigm for model-fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381–395. MathSciNetCrossRefGoogle Scholar
  7. Frankot, R. T., & Chellappa, R. (1988). A method for enforcing integrability in shape from shading algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 10(4), 439–451. zbMATHCrossRefGoogle Scholar
  8. Furukawa, Y., & Ponce, J. (2009). Dense 3d motion capture for human faces. In IEEE conference on computer vision and pattern recognition. Google Scholar
  9. Hernández, C., & Schmitt, F. (2004). Silhouette and stereo fusion for 3d object modeling. Computer Vision and Image Understanding, 96(3), 367–392. CrossRefGoogle Scholar
  10. Hernández, C., & Vogiatzis, G. (2010). Self-calibrating a real-time monocular 3d facial capture system. In Proceedings international symposium on 3D data processing, visualization and transmission (3DPVT) (p. 2010). Google Scholar
  11. Hernández, C., Vogiatzis, G., Brostow, G., Stenger, B., & Cipolla, R. (2007). Non-rigid photometric stereo with colored lights. In IEEE international conference on computer vision. Google Scholar
  12. Hernández, C., Vogiatzis, G., & Cipolla, R. (2008). Multi-view photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(3), 548–554. CrossRefGoogle Scholar
  13. Hernández, C., Vogiatzis, & Cipolla, R. (2008). Shadows in three-source photometric stereo. In IEEE European conference on computer vision. Google Scholar
  14. Image Metrics, U.I.f.C.T.: Emily project. In: SIGGRAPH 2008 demo session (2002). Google Scholar
  15. Kim, H., Wilburn, B., & Ben-Ezra, M. (2010). Photometric stereo for dynamic surface orientations. In Proc. European conf. on computer vision. Google Scholar
  16. Lin, Y., Medioni, G., & Choi, J. (2010). Accurate 3d face reconstruction from weakly calibrated wide baseline images with profile contours. In Computer vision and pattern recognition (CVPR). 2010 IEEE conference on (pp. 1490–1497). doi: 10.1109/CVPR.2010.5539793. CrossRefGoogle Scholar
  17. Ma, W., Hawkins, T., Peers, P., Chabert, C., Weiss, M., & Debevec, P. (2007). Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Eurographics symposium on rendering (pp. 183–194). Google Scholar
  18. Ma, W., Jones, A., Chiang, J., Hawkins, T., Frederiksen, S., Peers, P., Vukovic, M., Ouhyoung, M., & Debevec, P. Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Transactions on Graphics 27(5). Google Scholar
  19. Nehab, D., Rusinkiewicz, S., Davis, J., & Ramamoorthi, R. (2005). Efficiently combining positions and normals for precise 3d geometry. In Proc. of the ACM SIGGRAPH (pp. 536–543). Google Scholar
  20. Petrov, A. (1987). Light, color and shape. In Cognitive processes and their simulation (pp. 350–358) (in Russian). Google Scholar
  21. Seitz, S., Curless, B., Diebel, J., Scharstein, D., & Szeliski, R. (2006). A comparison and evaluation of multi-view stereo reconstruction algorithms. In Proc. IEEE conf. on computer vision and pattern recognition (pp. 519–528). Google Scholar
  22. Weise, T., Leibe, B., & Gool, L. V. (2007). Fast 3d scanning with automatic motion compensation. In IEEE conference on computer vision and pattern recognition. Google Scholar
  23. Woodham, R. (1980). Photometric method for determining surface orientation from multiple images. Optical Engineering, 19(1), 139–144. Google Scholar
  24. Woodham, R. J. (1994). Gradient and curvature from the photometric-stereo method, including local confidence estimation. Journal of the Optical Society of America, 11(11), 3050–3068. CrossRefGoogle Scholar
  25. Zhang, L., Snavely, N., Curless, B., & Seitz, S. M. (2004). Spacetime faces: high resolution capture for modeling and animation. In SIGGRAPH ’04 (pp. 548–558). CrossRefGoogle Scholar
  26. Zhang, S., & Huang, P. S. (2006). High-resolution, real-time three-dimensional shape measurement. Optical Engineering, 45(12), 123601. CrossRefGoogle Scholar
  27. Zisserman, A., & Hartley, R. (2000). Multiple view geometry. Berlin: Springer. zbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.Aston UniversityBirminghamUK
  2. 2.GoogleSeattleUS

Personalised recommendations