Skip to main content
SpringerLink
Log in

Neural style transfer combined with EfficientDet for thermal surveillance

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Hindrance caused while performing object detection during border or perimeter surveillance at night due to low resolution and atmospheric noise inside infrared frames, demands for a sophisticated detection framework. To deal with the challenges, we present a novel deep learning framework dedicated completely for automated thermal surveillance. The framework is a composition of two major modules. A novel style transferred enhanced image module STEIM enhances the resolution of IR input frame by utilizing the contextual information and exploiting the local and global features to preserve the high frequency details. The enhanced IR frame is further fed inside the fine-tuned EfficientDet module EDM that comprises weighted two-way feature network. It has an advantage of effectual multi-scale feature fusion contributing to accurate and efficient object detection. We have obtained mAP of 92.83% with 94.07% accuracy running at 95fps on FLIR test frames and mAP of 87.51% with 88.53% accuracy running at 89fps on OTCVBS test frames. Experimental results on two benchmark datasets FLIR and OTCBVS completely demonstrate suitability of the proposed framework for automatic surveillance using thermal imaging systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Availability of data and material

The authors declare that no data or material was taken illegally. However, publically available benchmark datasets were taken for implementation.

Code availability

The authors declare that no exact code has been copied to carry out the research.

References

  1. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2015)

    Article  Google Scholar 

  2. Zhang, H., Luo, C., Wang, Q., Kitchin, M., Parmley, A., Monge-Alvarez, J., Casaseca-De-La-Higuera, P.: A novel infrared video surveillance system using deep learning based techniques. Multimed. Tools Appl. 77(20), 26657–26676 (2018)

    Article  Google Scholar 

  3. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)

  4. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690 (2017)

  5. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Change Loy, C.: Esrgan: enhanced super-resolution generative adversarial networks. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018).

  6. Nasrollahi, K., Moeslund, T.B.: Super-resolution: a comprehensive survey. Mach. Vis. Appl. 25(6), 1423–1468 (2014)

    Article  Google Scholar 

  7. Yue, L., Shen, H., Li, J., Yuan, Q., Zhang, H., Zhang, L.: Image super-resolution: The techniques, applications, and future. Signal Process. 128, 389–408 (2016)

    Article  Google Scholar 

  8. Irani, M., Peleg, S.: Improving resolution by image registration. CVGIP Graph. Models Image Process. 53(3), 231–239 (1991)

    Article  Google Scholar 

  9. Tekalp, A.M., Ozkan, M.K., Sezan, M.I.: High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 3(3), pp. 169–172 (1992)

  10. Nguyen, N., Milanfar, P., Golub, G.: A computationally efficient super-resolution image reconstruction algorithm. IEEE Trans. Image Process. 10(4), 573–583 (2001)

    Article  MATH  Google Scholar 

  11. Tong, T., Li, G., Liu, X., Gao, Q.: Image super-resolution using dense skip connections. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4799–4807 (2017)

  12. Liu, J., Zou, M., Tang, J., Wu, G.: Memory recursive network for single image super-resolution. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 2202–2210 (2020)

  13. Xin, J., Jiang, X., Wang, N., Li, J., Gao, X.: Image super-resolution via deep feature recalibration network. In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pp. 256–267. Springer, Cham (2020)

  14. Yang, Y., Wang, X., Gao, X., Hui, Z.: Lightweight image super-resolution with local attention enhancement. In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pp. 219–231. Springer, Cham (2020)

  15. Mahapatra, D., Bozorgtabar, B., Garnavi, R.: Image super-resolution using progressive generative adversarial networks for medical image analysis. Comput. Med. Imaging Graph. 71, 30–39 (2019)

    Article  Google Scholar 

  16. Tian, C., Zhuge, R., Wu, Z., Xu, Y., Zuo, W., Chen, C., Lin, C.W.: Lightweight image super-resolution with enhanced CNN. Knowl. Based Syst. 205, 106235 (2020)

    Article  Google Scholar 

  17. Ashiba, H.I., Mansour, H.M., Ahmed, H.M., El-Kordy, M.F., Dessouky, M.I., Abd El-Samie, F.E.: Enhancement of infrared images based on efficient histogram processing. Wireless Pers. Commun. 99(2), 619–636 (2018)

    Article  Google Scholar 

  18. Bhattacharya, S., Idrees, H., Saleemi, I., Ali, S., Shah, M.: Moving object detection and tracking in forward looking infra-red aerial imagery. In: Machine Vision Beyond Visible Spectrum, pp. 221–252. Springer, Berlin, Heidelberg (2011)

  19. Liu, F., Han, P., Wang, Y., Li, X., Bai, L., Shao, X.: Super resolution reconstruction of infrared images based on classified dictionary learning. Infrared Phys. Technol. 90, 146–155 (2018)

    Article  Google Scholar 

  20. Zhao, Y., Chen, Q., Sui, X., Gu, G.: A novel infrared image super-resolution method based on sparse representation. Infrared Phys. Technol. 71, 506–513 (2015)

    Article  Google Scholar 

  21. Chen, Y., Shin, H.: Pedestrian detection at night in infrared images using an attention-guided encoder-decoder convolutional neural network. Appl. Sci. 10(3), 809 (2020)

    Article  Google Scholar 

  22. Biswas, S.K., Milanfar, P.: Linear support tensor machine with LSK channels: pedestrian detection in thermal infrared images. IEEE Trans. Image Process. 26(9), 4229–4242 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  23. Algarni, A.D.: Efficient object detection and classification of heat emitting objects from infrared images based on deep learning. Multimedia Tools and Applications, pp. 1–24 (2020)

  24. Rodin, C.D., de Lima, L.N., de Alcantara Andrade, F.A., Haddad, D.B., Johansen, T.A., Storvold, R.: Object classification in thermal images using convolutional neural networks for search and rescue missions with unmanned aerial systems. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2018)

  25. Lile, C., Yiqun, L.: Anomaly detection in thermal images using deep neural networks. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 2299–2303. IEEE (2017)

  26. Dai, X., Yuan, X., Wei, X.: TIRNet: object detection in thermal infrared images for autonomous driving. Appl. Intell., pp. 1–18 (2020)

  27. Wang, Z., Chen, J., Hoi, S.C.: Deep learning for image super-resolution: a survey. IEEE Trans. Pattern Anal. Mach. Intell. (2020)

  28. Davis, J.W., Sharma, V.: Background-subtraction using contour-based fusion of thermal and visible imagery. Comput. Vis. Image Underst. 106(2–3), 162–182 (2007)

    Article  Google Scholar 

  29. Tan, M., Pang, R., Le, Q.V.: Efficientdet: scalable and efficient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10781–10790 (2020)

  30. Goyal, P., Kaiming, H.: Focal loss for dense object detection. IEEE Trans. Pattern Anal. Mach. Intell. 39, 2999–3007 (2018)

    Google Scholar 

  31. Wang, K., Liew, J.H., Zou, Y., Zhou, D., Feng, J.: Panet: few-shot image semantic segmentation with prototype alignment. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 9197–9206 (2019).

  32. Adelson, E.H., Anderson, C.H., Bergen, J.R., Burt, P.J., Ogden, J.M.: Pyramid methods in image processing. RCA Eng. 29(6), 33–41 (1984)

    Google Scholar 

  33. Tan, M., Le, Q.V.: Efficientnet: Rethinking Model Scaling for Convolutional Neural Networks (2019). arXiv preprint https://arxiv.org.abs/1905.11946

  34. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: Ssd: single shot multibox detector. In: European conference on computer vision, pp. 21–37. Springer, Cham (2016).

  35. Deng, X.: Enhancing image quality via style transfer for single image super-resolution. IEEE Signal Process. Lett. 25(4), 571–575 (2018)

    Article  Google Scholar 

  36. Gatys, L.A., Ecker, A.S., Bethge, M.: A Neural Algorithm of Artistic Style (2015). arXiv preprint https://arxiv.org/abs/1508.06576

  37. Gong, J., Zhao, J., Li, F., Zhang, H.: Vehicle detection in thermal images with an improved yolov3-tiny. In: 2020 IEEE International Conference on Power, Intelligent Computing and Systems (ICPICS), pp. 253–256. IEEE (2020)

  38. Munir, F., Azam, S., Rafique, M.A., Sheri, A.M., Jeon, M. Thermal Object Detection using Domain Adaptation through Style Consistency (2020). arXiv preprint https://arxiv.org/abs/2006.00821

  39. Devaguptapu, C., Akolekar, N., M Sharma, M., Balasubramanian, V.: Borrow from anywhere: pseudo multi-modal object detection in thermal imagery. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2019)

  40. Zhang, H., Fromont, E., Lefèvre, S., Avignon, B.: Multispectral fusion for object detection with cyclic fuse-and-refine blocks. In: 2020 IEEE International Conference on Image Processing (ICIP), pp. 276–280. IEEE (2020)

  41. Cheng, G., Zhou, P., Han, J.: Learning rotation-invariant convolutional neural networks for object detection in VHR optical remote sensing images. IEEE Trans. Geosci. Remote Sens. 54(12), 7405–7415 (2016)

    Article  Google Scholar 

  42. Ravishankar, M., Rameshbabu, D.R.: Ten-LoPP: tensor locality preserving projections approach for moving object detection and tracking. In: The 9th international conference on computing and information technology (IC2IT2013), pp. 291–300. Springer, Berlin, Heidelberg (2013)

  43. Wang, J., Zhang, Y., Lu, J., Li, Y.: Target detection and pedestrian recognition in infrared images. JCP 8(4), 1050–1057 (2013)

    Google Scholar 

  44. Lisin, D.A., Mattar, M.A., Blaschko, M.B., Learned-Miller, E.G., Benfield, M.C.: Combining local and global image features for object class recognition. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05)-Workshops, pp. 47–47. IEEE (2005)

  45. Zhang, X., Feng, R., Li, X., Shen, H., Yuan, Z.: Block adjustment-based radiometric normalization by considering global and local differences. IEEE Geosci. Remote Sens. Lett. (2020)

  46. Li, S., Jin, W., Li, L., Li, Y.: An improved contrast enhancement algorithm for infrared images based on adaptive double plateaus histogram equalization. Infrared Phys. Technol. 90, 164–174 (2018)

    Article  Google Scholar 

  47. Shakya, A.K., Ramola, A., Vidyarthi, A., Sawant, K.: satellite image enhancement for small particle observation using decorrelation stretcher. In: 2020 International Conference on Advances in Computing, Communication and Materials (ICACCM), pp. 65–70. IEEE (2020)

  48. Tang, H., Ni, R., Zhao, Y., Li, X.: Median filtering detection of small-size image based on CNN. J. Vis. Commun. Image Represent. 51, 162–168 (2018)

    Article  Google Scholar 

  49. Jagatheeswari, P., Kumar, S.S., Rajaram, M.: Contrast stretching recursively separated histogram equalization for brightness preservation and contrast enhancement. In: 2009 International Conference on Advances in Computing, Control, and Telecommunication Technologies, pp. 111–115. IEEE (2009)

  50. Yang, T., Liang, R., Huang, L.: Vehicle counting method based on attention mechanism SSD and state detection. Vis. Comput., pp. 1–11 (2021)

  51. Ahmad, N., Asghar, S., Gillani, S.A.: Transfer learning-assisted multi-resolution breast cancer histopathological images classification. Vis. Comput., pp. 1–20 (2021)

  52. Abbass, M.Y., Kwon, K.C., Kim, N., Abdelwahab, S.A., Abd El-Samie, F.E., Khalaf, A.A.: Efficient object tracking using hierarchical convolutional features model and correlation filters. Vis. Comput. 37(4), 831–842 (2021)

    Article  Google Scholar 

Download references

Funding

The authors declare that no funding was received for this research work.

Author information

Authors and Affiliations

  1. Department of Electronics and Communication Engineering, Punjab Engineering College, Sector-12, Chandigarh, India

    Aayushi Gautam & Sukhwinder Singh

Authors
  1. Aayushi Gautam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sukhwinder Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aayushi Gautam.

Ethics declarations

Conflict of interest

The authors declare that there are no conflicts of interest in this research.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gautam, A., Singh, S. Neural style transfer combined with EfficientDet for thermal surveillance. Vis Comput 38, 4111–4127 (2022). https://doi.org/10.1007/s00371-021-02284-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02284-2

Keywords

Search

Navigation