Super-Resolved Faces for Improved Face Recognition from Surveillance Video

  • Frank Lin
  • Clinton Fookes
  • Vinod Chandran
  • Sridha Sridharan
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4642)


Characteristics of surveillance video generally include low resolution and poor quality due to environmental, storage and processing limitations. It is extremely difficult for computers and human opera- tors to identify individuals from these videos. To overcome this problem, super-resolution can be used in conjunction with an automated face recognition system to enhance the spatial resolution of video frames containing the subject and narrow down the number of manual verifications performed by the human operator by presenting a list of most likely candidates from the database. As the super-resolution reconstruction process is ill-posed, visual artifacts are often generated as a result. These artifacts can be visually distracting to humans and/or affect machine recognition algorithms. While it is intuitive that higher resolution should lead to improved recognition accuracy, the effects of super-resolution and such artifacts on face recognition performance have not been systematically studied. This paper aims to address this gap while illustrating that super-resolution allows more accurate identification of individuals from low-resolution surveillance footage. The proposed optical flow-based super-resolution method is benchmarked against Baker et al.’s hallucination and Schultz et al.’s super-resolution techniques on images from the Terrascope and XM2VTS databases. Ground truth and interpolated images were also tested to provide a baseline for comparison. Results show that a suitable super-resolution system can improve the discriminability of surveillance video and enhance face recognition accuracy. The experiments also show that Schultz et al.’s method fails when dealing surveillance footage due to its assumption of rigid objects in the scene. The hallucination and optical flow-based methods performed comparably, with the optical flow-based method producing less visually distracting artifacts that interfered with human recognition.


super-resolution face recognition surveillance 


  1. 1.
    Gunturk, B., Batur, A., Altunbasak, Y., III, M.H., Mersereau, R.: Eigenface-domain super-resolution for face recognition. IEEE Transactions on Image Processing 12(5), 597–606 (2003)CrossRefGoogle Scholar
  2. 2.
    Lemieux, A., Parizeau, M.: Experiments on eigenfaces robustness. In: Proc. ICPR-2002, vol. 1, pp. 421–424 (August 2002)Google Scholar
  3. 3.
    Wang, X., Tang, X.: Face Hallucination and Recognition. In: Kittler, J., Nixon, M.S. (eds.) AVBPA 2003. LNCS, vol. 2688, pp. 486–494. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  4. 4.
    Jaynes, C., Kale, A., Sanders, N., Grossmann, E.: The Terrascope dataset: scripted multi-camera indoor video surveillance with ground-truth. In: Proc. Visual Surveillance and Performance Evaluation of Tracking and Surveillance, pp. 309–316 (October 2005)Google Scholar
  5. 5.
    Turk, M., Pentland, A.: Eigenfaces for recognition. Journal of Cognitive Neuroscience 3(1), 71–86 (1991)CrossRefGoogle Scholar
  6. 6.
    Wiskott, L., Fellous, J., Krüger, N., Malsburg, C.: Face recognition by elastic bunch graph matching. In: Sommer, G., Daniilidis, K., Pauli, J. (eds.) CAIP 1997. LNCS, vol. 1296, pp. 456–463. Springer, Heidelberg (1997)CrossRefGoogle Scholar
  7. 7.
    Messer, K., Matas, J., Kittler, J., Luettin, J., Maitre, G.: XM2VTS: The Extended M2VTS Database. In: Proc. AVBPA-1999, pp. 72–76 (1999)Google Scholar
  8. 8.
    Park, S., Park, M., Kang, M.: Super-resolution image reconstruction: a technical overview. IEEE Signal Processing Magazine 25(9), 21–36 (2003)CrossRefGoogle Scholar
  9. 9.
    Tsai, R., Huang, T.: Multiframe image restoration and registration. Advances in Computer Vision and image Processing 1, 317–339 (1984)Google Scholar
  10. 10.
    Baker, S., Kanade, T.: Limits on Super-Resolution and How to Break Them. 24(9), 1167–1183 (2002)Google Scholar
  11. 11.
    Baker, S., Kanade, T.: Super Resolution Optical Flow. Technical Report CMU-RI-TR-99-36, The Robotics Institute, Carnegie Mellon University (October 1999)Google Scholar
  12. 12.
    Lin, F., Fookes, C., Chandran, V., Sridharan, S.: Investigation into Optical Flow Super-Resolution for Surveillance Applications. In: Proc. APRS Workshop on Digital Image Computing 2005, pp. 73–78 (February 2005)Google Scholar
  13. 13.
    Black, M., Anandan, P.: A framework for the robust estimation of optical flow. In: Proc. ICCV-1993, pp. 231–236 (May 1993)Google Scholar
  14. 14.
    Schultz, R., Stevenson, R.: Extraction of High-Resolution Frames from Video Sequences. IEEE Transactions on Image Processing 5(6), 996–1011 (1996)CrossRefGoogle Scholar
  15. 15.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: CVPR (2001)Google Scholar
  16. 16.
    Bolme, D., Beveridge, R., Teixeira, M., Draper, B.: The CSU Face Identification Evaluation System: Its Purpose, Features and Structure. In: Proc. International Conference on Vision Systems, pp. 304–311 (April 2003)Google Scholar
  17. 17.
    Phillips, P., Flynn, P., Scruggs, T., Bowyer, K., Chang, J., Hoffman, K., Marques, J., Min, J., Worek, W.: Overview of the face recognition grand challenge. In: Proc. CVPR 2005, vol. 1, pp. 947–954 (2005)Google Scholar
  18. 18.
    Phillips, P., Moon, H., Rizvi, S., Rauss, P.: The feret evaluation methodology for face-recognition algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 1090–1104 (2000)CrossRefGoogle Scholar
  19. 19.
    Lin, F., Cook, J., Chandran, V., Sridharan, S.: Face Recognition from Super-Resolved Images. In: Proc. ISSPA 2005, pp. 667–670 (August 2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Frank Lin
    • 1
  • Clinton Fookes
    • 1
  • Vinod Chandran
    • 1
  • Sridha Sridharan
    • 1
  1. 1.Image and Video Research Laboratory, Queensland University of Technology, GPO Box 2434 Brisbane, QLD 4001Australia

Personalised recommendations