Skip to main content
SpringerLink
Log in

Deep Perceptual Mapping for Cross-Modal Face Recognition

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Cross modal face matching between the thermal and visible spectrum is a much desired capability for night-time surveillance and security applications. Due to a very large modality gap, thermal-to-visible face recognition is one of the most challenging face matching problem. In this paper, we present an approach to bridge this modality gap by a significant margin. Our approach captures the highly non-linear relationship between the two modalities by using a deep neural network. Our model attempts to learn a non-linear mapping from the visible to the thermal spectrum while preserving the identity information. We show substantive performance improvement on three difficult thermal–visible face datasets. The presented approach improves the state-of-the-art by more than 10 % on the UND-X1 dataset and by more than 15–30 % on the NVESD dataset in terms of Rank-1 identification. Our method bridges the drop in performance due to the modality gap by more than 40 %.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  • Bourlai, T., Ross, A., Chen, C., & Hornak, L. (2012). A study on using mid-wave infrared images for face recognition. In SPIE Defense, Security, and Sensing, International Society for Optics and Photonics (pp 83,711K–83,711K).

  • Byrd, K. (2013). Preview of the newly acquired nvesd-arl multimodal face database. In: Proceedings of SPIE, vol 8734 (pp. 8734–8734).

  • Chen, C., & Ross, A. (2015). Matching thermal to visible face images using hidden factor analysis in a cascaded subspace learning framework. Pattern Recognition Letters. doi:10.1016/j.patrec.2015.06.021. http://www.sciencedirect.com/science/article/pii/S0167865515001932.

  • Chen, X., Flynn, P. J., & Bowyer, K. W. (2005). IR and visible light face recognition. Computer Vision and Image Understanding, 99(3), 332–358.

    Article  Google Scholar 

  • Choi, J., Hu, S., Young, S. S., & Davis, L. S. (2012). Thermal to visible face recognition. In SPIE Defense, Security, and Sensing, International Society for Optics and Photonics (pp. 83,711L–83,711L).

  • Espinosa-Dur, V., Faundez-Zanuy, M., & Mekyska, J. (2013). A new face database simultaneously acquired in visible, near-infrared and thermal spectrums. Cognitive Computation, 5(1), 119–135. doi:10.1007/s12559-012-9163-2.

    Article  Google Scholar 

  • Ganin, Y., & Lempitsky, V. (2014). Unsupervised domain adaptation by backpropagation. arXiv:1409.7495.

  • Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In International conference on artificial intelligence and statistics (pp 249–256).

  • Hu, J., Lu, J., & Tan, Y.P. (2014a). Discriminative deep metric learning for face verification in the wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1875–1882).

  • Hu, S., Gurram, P., Kwon, H., & Chan, A. L. (2014b). Thermal-to-visible face recognition using multiple kernel learning. In SPIE Defense+ Security, International Society for Optics and Photonics (pp 909,110–909,110).

  • Hu, S., Choi, J., Chan, A. L., & Schwartz, W. R. (2015). Thermal-to-visible face recognition using partial least squares. Journal of the Optical Society of America, 32(3), 431–442.

    Article  Google Scholar 

  • Klare, B., & Jain, A. (2010). Heterogeneous face recognition: Matching nir to visible light images. In 20th International Conference on Pattern Recognition (ICPR), 2010 (pp. 1513–1516).

  • Klare, B. F., & Jain, A. K. (2013). Heterogeneous face recognition using kernel prototype similarities. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(6), 1410–1422.

    Article  Google Scholar 

  • Lei, Z., & Li, S. Z. (2009). Coupled spectral regression for matching heterogeneous faces. In IEEE Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009 (pp 1123–1128).

  • Li, J., Hao, P., Zhang, C., & Dou, M. (2008). Hallucinating faces from thermal infrared images. In 15th IEEE International Conference on Image Processing, 2008. ICIP 2008 (pp. 465–468).

  • Li, S. Z., Chu, R., Liao, S., & Zhang, L. (2007). Illumination invariant face recognition using near-infrared images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(4), 627–639.

    Article  Google Scholar 

  • Liao, S., Yi, D., Lei, Z., Qin, R., & Li, S. Z. (2009). Heterogeneous face recognition from local structures of normalized appearance. In: Advances in Biometrics, Springer (pp. 209–218).

  • Mostafa, E., Hammoud, R., Ali, A., & Farag, A. (2013). Face recognition in low resolution thermal images. Computer Vision and Image Understanding, 117(12), 1689–1694.

    Article  Google Scholar 

  • Nicolo, F., & Schmid, N. A. (2012). Long range cross-spectral face recognition: Matching swir against visible light images. IEEE Transactions on Information Forensics and Security, 7(6), 1717–1726.

  • Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015). Deep face recognition. In British Machine Vision Conference.

  • Riggan, B. S., Nathaniel, J. S., & Shuowen, H. (2016). Optimal feature learning and discriminative framework for polarimetric thermal to visible face recognition. In IEEE Winter Conference on Applications of Computer Vision (WACV).

  • Rosipal, R., & Krämer, N. (2006). Overview and recent advances in partial least squares. In Subspace, latent structure and feature selection, Springer (pp. 34–51).

  • Ross, T. B. N. K. A., & Hornak, B. C. L. (2010). Cross-spectral face verification in the short wave infrared (swir) band.

  • Sarfraz, M. S., & Stiefelhagen, R. (2015). Deep perceptual mapping for thermal to visible face recognition. In British Machine Vision Conference.

  • Schroff, F., Kalenichenko, D., & Philbin, J. (2015). Facenet: A unified embedding for face recognition and clustering. arXiv:1503.03832.

  • Socolinsky, D. A., & Selinger, A. (2002). A comparative analysis of face recognition performance with visible and thermal infrared imagery. Tech. rep., DTIC Document.

  • Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014). Deepface: Closing the gap to human-level performance in face verification. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014 (pp. 1701–1708).

  • Yi, D., Liu, R., Chu, R., Lei, Z., & Li, S. Z. (2007). Face matching between near infrared and visible light images. In: Advances in Biometrics, Springer (pp 523–530).

  • Zhou, H., Mian, A., Wei, L., Creighton, D., Hossny, M., & Nahavandi, S. (2014). Recent advances on singlemodal and multimodal face recognition: A survey. IEEE Transactions on Human Machine Systems, 44(6), 701–716.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Anthropomatics & Robotics, Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany

    M. Saquib Sarfraz & Rainer Stiefelhagen

Authors
  1. M. Saquib Sarfraz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rainer Stiefelhagen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M. Saquib Sarfraz.

Additional information

Communicated by Xianghua Xie, Mark Jones and Gary Tam.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sarfraz, M.S., Stiefelhagen, R. Deep Perceptual Mapping for Cross-Modal Face Recognition. Int J Comput Vis 122, 426–438 (2017). https://doi.org/10.1007/s11263-016-0933-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-016-0933-2

Keywords

Search

Navigation