Multimodal Sensor Fusion in Single Thermal Image Super-Resolution

  • Feras Almasri
  • Olivier DebeirEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11367)


With the fast growth in the visual surveillance and security sectors, thermal infrared images have become increasingly necessary in a large variety of industrial applications. This is true even though IR sensors are still more expensive than their RGB counterpart having the same resolution. In this paper, we propose a deep learning solution to enhance the thermal image resolution. The following results are given: (I) Introduction of a multimodal, visual-thermal fusion model that addresses thermal image super-resolution, via integrating high-frequency information from the visual image. (II) Investigation of different network architecture schemes in the literature, their up-sampling methods, learning procedures, and their optimization functions by showing their beneficial contribution to the super-resolution problem. (III) A benchmark ULB17-VT dataset that contains thermal images and their visual images counterpart is presented. (IV) Presentation of a qualitative evaluation of a large test set with 58 samples and 22 raters which shows that our proposed model performs better against state-of-the-arts.


Super-resolution Sensor fusion Thermal images 


  1. 1.
    Chen, X., Zhai, G., Wang, J., Hu, C., Chen, Y.: Color guided thermal image super resolution. In: Visual Communications and Image Processing (VCIP), pp. 1–4. IEEE (2016)Google Scholar
  2. 2.
    Chen, Z., Tong, Y.: Face super-resolution through Wasserstein GANs. arXiv preprint arXiv:1705.02438 (2017)
  3. 3.
    Cho, H., Seo, Y.W., Kumar, B.V., Rajkumar, R.R.: A multi-sensor fusion system for moving object detection and tracking in urban driving environments. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 1836–1843. IEEE (2014)Google Scholar
  4. 4.
    Choi, Y., Kim, N., Hwang, S., Kweon, I.S.: Thermal image enhancement using convolutional neural network. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 223–230. IEEE (2016)Google Scholar
  5. 5.
    Clevert, D.A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (ELUs). arXiv preprint arXiv:1511.07289 (2015)
  6. 6.
    Dong, C., Change Loy, C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. ArXiv e-prints, December 2015Google Scholar
  7. 7.
    Goodfellow, I.: NIPS 2016 tutorial: generative adversarial networks. arXiv preprint arXiv:1701.00160 (2016)
  8. 8.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  9. 9.
    Gross, S., Wilber, M.: Training and investigating residual nets (2016).
  10. 10.
    Huang, Y., Qin, M.: Densely connected high order residual network for single frame image super resolution. arXiv preprint arXiv:1804.05902 (2018)
  11. 11.
    Hwang, S., Park, J., Kim, N., Choi, Y., Kweon, I.S.: Multispectral pedestrian detection: benchmark dataset and baselines. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  12. 12.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)
  13. 13.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. arXiv preprint (2017)Google Scholar
  14. 14.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). Scholar
  15. 15.
    Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)Google Scholar
  16. 16.
    Kim, J., Kwon Lee, J., Mu Lee, K.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1637–1645 (2016)Google Scholar
  17. 17.
    Kiran, Y., Shrinidhi, V., Hans, W.J., Venkateswaran, N.: A single-image super-resolution algorithm for infrared thermal images. Int. J. Comput. Sci. Netw. Secur. 17(10), 256–261 (2017)Google Scholar
  18. 18.
    Lai, W.S., Huang, J.B., Ahuja, N., Yang, M.H.: Deep Laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings IEEE Conference Computer Vision and Pattern Recognition, pp. 624–632 (2017)Google Scholar
  19. 19.
    Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. ArXiv e-prints, September 2016Google Scholar
  20. 20.
    Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint (2016)Google Scholar
  21. 21.
    Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, vol. 1, p. 4 (2017)Google Scholar
  22. 22.
    Panagiotopoulou, A., Anastassopoulos, V.: Super-resolution reconstruction of thermal infrared images. In: Proceedings of the 4th WSEAS International Conference on Remote Sensing (2008)Google Scholar
  23. 23.
    Qu, Y., Zhang, G., Zou, Z., Liu, Z., Mao, J.: Active multimodal sensor system for target recognition and tracking. Sensors 17(7), 1518 (2017)CrossRefGoogle Scholar
  24. 24.
    Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)Google Scholar
  25. 25.
    Wu, B., Duan, H., Liu, Z., Sun, G.: SRPGAN: perceptual generative adversarial network for single image super resolution. arXiv preprint arXiv:1712.05927 (2017)
  26. 26.
    Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. ArXiv e-prints, November 2016Google Scholar
  27. 27.
    Zhang, H., et al.: StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks. In: IEEE International Conference on Computer Vision (ICCV), pp. 5907–5915 (2017)Google Scholar
  28. 28.
    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint arXiv:1703.10593 (2017)

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department LISA - Laboratory of Image Synthesis and AnalysisUniversité Libre de BruxellesBrusselsBelgium

Personalised recommendations