Illumination-Invariant Face Recognition by Fusing Thermal and Visual Images via Gradient Transfer

  • Sumit AgarwalEmail author
  • Harshit S. Sikchi
  • Suparna Rooj
  • Shubhobrata Bhattacharya
  • Aurobinda Routray
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 943)


Face recognition in real life situations like low illumination condition is still an open challenge in biometric security. It is well established that the state-of-the-art methods in face recognition provide low accuracy in the case of poor illumination. In this work, we propose an algorithm for a more robust illumination invariant face recognition using a multi-modal approach. We propose a new dataset consisting of aligned faces of thermal and visual images of a hundred subjects. We then apply face detection on thermal images using the biggest blob extraction method and apply them for fusing images of different modalities for the purpose of face recognition. An algorithm is proposed to implement fusion of thermal and visual images. We reason for why relying on only one modality can give erroneous results. We use a lighter and faster CNN model called MobileNet for the purpose of face recognition with faster inferencing and to be able to use it in real time biometric systems. We test our proposed method on our own created dataset to show that real-time face recognition on fused images shows far better results than using visual or thermal images separately.


Biometrics Face recognition Image fusion Thermal face detection Gradient transfer MobileNet 


  1. 1.
    Ekenel, H.K., Stallkamp, J., Gao, H., Fischer, M., Stiefelhagen, R.: Face recognition for smart interactions. In: 2007 IEEE International Conference on Multimedia and Expo, pp. 1007–1010. IEEE (2007)Google Scholar
  2. 2.
    Pentland, A., Choudhury, T.: Face recognition for smart environments. Computer 33(2), 50–55 (2000)CrossRefGoogle Scholar
  3. 3.
    Galton, F.: Personal identification and description. J. Anthropol. Inst. Great Br. Irel. 18, 177–191 (1889)CrossRefGoogle Scholar
  4. 4.
    Belhumeur, P.N., Hespanha, J.P., Kriegman, D.J.: Eigenfaces vs. fisherfaces: recognition using class specific linear projection. Yale University New Haven United States, Technical report (1997)CrossRefGoogle Scholar
  5. 5.
    He, X., Yan, S., Hu, Y., Niyogi, P., Zhang, H.-J.: Face recognition using Laplacianfaces. IEEE Trans. Pattern Anal. Mach. Intell. 27(3), 328–340 (2005)CrossRefGoogle Scholar
  6. 6.
    Gao, Y., Leung, M.K.: Face recognition using line edge map. IEEE Trans. Pattern Anal. Mach. Intell. 24, 764–779 (2002)CrossRefGoogle Scholar
  7. 7.
    Kirby, M., Sirovich, L.: Application of the Karhunen-Loeve procedure for the characterization of human faces. IEEE Trans. Pattern Anal. Mach. Intell. 12(1), 103–108 (1990)CrossRefGoogle Scholar
  8. 8.
    Bartlett, M.S., Movellan, J.R., Sejnowski, T.J.: Face recognition by independent component analysis. IEEE Trans. Neural Netw. 13(6), 1450 (2002)CrossRefGoogle Scholar
  9. 9.
    Adini, Y., Moses, Y., Ullman, S.: Face recognition: the problem of compensating for changes in illumination direction. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 721–732 (1997)CrossRefGoogle Scholar
  10. 10.
    Wen, D., Han, H., Jain, A.K.: Face spoof detection with image distortion analysis. IEEE Trans. Inf. Forensics Secur. 10(4), 746–761 (2015)CrossRefGoogle Scholar
  11. 11.
    Cutler, R.G.: Face recognition using infrared images and eigenfaces. University of Maryland (1996)Google Scholar
  12. 12.
    Bebis, G., Gyaourova, A., Singh, S., Pavlidis, I.: Face recognition by fusing thermal infrared and visible imagery. Image Vis. Comput. 24(7), 727–742 (2006)CrossRefGoogle Scholar
  13. 13.
    Socolinsky, D.A., Selinger, A., Neuheisel, J.D.: Face recognition with visible and thermal infrared imagery. Comput. Vis. Image Underst. 91(1–2), 72–114 (2003)CrossRefGoogle Scholar
  14. 14.
    Forczmański, P.: Human face detection in thermal images using an ensemble of cascading classifiers. In: International Multi-Conference on Advanced Computer Systems, pp. 205–215. Springer (2016)Google Scholar
  15. 15.
    Wong, W.K., Hui, J.H., Desa, J.B.M., Ishak, N.I.N.B., Sulaiman, A.B., Nor, Y.B.M.: Face detection in thermal imaging using head curve geometry. In: 5th International Congress on Image and Signal Processing (CISP), pp. 881–884. IEEE (2012)Google Scholar
  16. 16.
    Selinger, A., Socolinsky, D.A.: Appearance-based facial recognition using visible and thermal imagery: a comparative study. Technical report, EQUINOX CORP NEW YORK NY (2006)Google Scholar
  17. 17.
    Nguyen, H., Kotani, K., Chen, F., Le, B.: A thermal facial emotion database and its analysis. In: Pacific-Rim Symposium on Image and Video Technology, pp. 397–408. Springer (2013)Google Scholar
  18. 18.
    Wang, S., Liu, Z., Lv, S., Lv, Y., Wu, G., Peng, P., Chen, F., Wang, X.: A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE Trans. Multimedia 12(7), 682–691 (2010)CrossRefGoogle Scholar
  19. 19.
    Ma, J., Chen, C., Li, C., Huang, J.: Infrared and visible image fusion via gradient transfer and total variation minimization. Inf. Fusion 31, 100–109 (2016)CrossRefGoogle Scholar
  20. 20.
    Chan, T.F., Esedoglu, S.: Aspects of total variation regularized \(L^1\) function approximation. SIAM J. Appl. Math. 65(5), 1817–1837 (2005)MathSciNetCrossRefGoogle Scholar
  21. 21.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436 (2015)CrossRefGoogle Scholar
  22. 22.
    Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: efficient convolutional neural networks for mobile vision applications, CoRR, vol. abs/1704.04861 (2017)Google Scholar
  23. 23.
    Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition, vol. 2, no. 6, arXiv preprintarXiv:1707.07012 (2017)Google Scholar
  24. 24.
    Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.-C.: MobileNetV2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Sumit Agarwal
    • 1
    Email author
  • Harshit S. Sikchi
    • 1
  • Suparna Rooj
    • 1
  • Shubhobrata Bhattacharya
    • 1
  • Aurobinda Routray
    • 1
  1. 1.Indian Institute of TechnologyKharagpurIndia

Personalised recommendations