Skip to main content
SpringerLink
Log in

Remote detection of idling cars using infrared imaging and deep networks

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Idling vehicles waste energy and pollute the environment through exhaust emission. In some countries, idling a vehicle for more than a predefined duration is prohibited and automatic idling vehicle detection is desirable for law enforcement. We propose the first automatic system to detect idling cars, using infrared (IR) imaging and deep networks. We rely on the differences in spatio-temporal heat signatures of idling and stopped cars and monitor the car temperature with a long-wavelength IR camera. We formulate the idling car detection problem as spatio-temporal event detection in IR image sequences and employ deep networks for spatio-temporal modeling. We collected the first IR image sequence dataset for idling car detection. First, we detect the cars in each IR image using a convolutional neural network, which is pre-trained on regular RGB images and fine-tuned on IR images for higher accuracy. Then, we track the detected cars over time to identify the cars that are parked. Finally, we use the 3D spatio-temporal IR image volume of each parked car as input to convolutional and recurrent networks to classify them as idling or not. We carried out an extensive empirical evaluation of temporal and spatio-temporal modeling approaches with various convolutional and recurrent architectures. We present promising experimental results on our IR image sequence dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Akhloufi M, Bendada A (2008) Thermal faceprint: a new thermal face signature extraction for infrared face recognition. In: Canadian conference on computer and robot vision, pp 269–272. IEEE

  2. Arlot S, Celisse A (2010) A survey of cross-validation procedures for model selection. Stat Surv 4:40–79

    Article  MathSciNet  Google Scholar 

  3. Azizpour H, Sharif Razavian A, Sullivan J, Maki A, Carlsson S (2015) From generic to specific deep representations for visual recognition. In: IEEE conference on computer vision and pattern recognition workshops, pp 36–45

  4. Bastan M, Yap KH, Chau LP (2018) Idling car detection with ConvNets in infrared image sequences. In: International symposium on circuits and systems. IEEE

  5. Bebis G, Gyaourova A, Singh S, Pavlidis I (2006) Face recognition by fusing thermal infrared and visible imagery. Image Vis Comput 24(7):727–742

    Article  Google Scholar 

  6. Bertozzi M, Broggi A, Caraffi C, Del Rose M, Felisa M, Vezzoni G (2007) Pedestrian detection by means of far-infrared stereo vision. Comput Vis Image Underst 106(2):194–204

    Article  Google Scholar 

  7. Bodansky D (2016) The Paris climate change agreement: a new hope? Am J Int Law 110(2):288–319

    Article  Google Scholar 

  8. Chen Y, Zhang X, Zhang Y, Maybank SJ, Fu Z (2018) Visible and infrared image registration based on region features and edginess. Mach Vis Appl 29(1):113–123

    Article  Google Scholar 

  9. Chollet F et al (2017) Keras. https://github.com/fchollet/keras. Accessed 2018

  10. Chung JS, Senior A, Vinyals O, Zisserman A (2017) Lip reading sentences in the wild. In: IEEE conference on computer vision and pattern recognition

  11. Damien A (2017) TFLearn. https://github.com/tflearn. Accessed 2018

  12. Donahue J, Anne Hendricks L, Guadarrama S, Rohrbach M, Venugopalan S, Saenko K, Darrell T (2015) Long-term recurrent convolutional networks for visual recognition and description. In: IEEE conference on computer vision and pattern recognition, pp 2625–2634

  13. Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The Pascal visual object classes (VOC) challenge. Int J Comput Vis 88(2):303–338

    Article  Google Scholar 

  14. Fendri E, Boukhriss RR, Hammami M (2017) Fusion of thermal infrared and visible spectra for robust moving object detection. Pattern Anal Appl 20:907–926

    Article  MathSciNet  Google Scholar 

  15. Filipe S, Alexandre LA (2014) Algorithms for invariant long-wave infrared face segmentation: evaluation and comparison. Pattern Anal Appl 17(4):823–837

    Article  MathSciNet  Google Scholar 

  16. Gade R, Moeslund TB (2014) Thermal cameras and applications: a survey. Mach Vis Appl 25(1):245–262

    Article  Google Scholar 

  17. Gaines L, Rask E, Keller G (2012) Which is greener: idle, or stop and restart. Argonne National Laboratory, US Department of Energy

  18. Gault T, Farag A (2013) A fully automatic method to extract the heart rate from thermal video. In: IEEE conference on computer vision and pattern recognition workshops, pp 336–341

  19. Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: International conference on artificial intelligence and statistics, pp 249–256

  20. Google (2017) TensorFlow. https://github.com/tensorflow. Accessed 2018

  21. Hinz S, Stilla U (2006) Car detection in aerial thermal images by local and global evidence accumulation. Pattern Recognit Lett 27(4):308–315

    Article  Google Scholar 

  22. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  23. Hou R, Chen C, Shah M (2017) Tube convolutional neural network (T-CNN) for action detection in videos. In: International conference on computer vision

  24. Huang J, Rathod V, Sun C, Zhu M, Korattikara A, Fathi A, Fischer I, Wojna Z, Song Y, Guadarrama S, Murphy K (2017) Speed/accuracy trade-offs for modern convolutional object detectors. In: IEEE conference on computer vision and pattern recognition, vol 4

  25. Keskar NS, Socher R (2017) Improving Generalization performance by switching from Adam to SGD. arXiv preprint arXiv:1712.07628

  26. Kim S (2014) Analysis of small infrared target features and learning-based false detection removal for infrared search and track. Pattern Anal Appl 17(4):883–900

    Article  MathSciNet  Google Scholar 

  27. Kingma D, Ba J (2014) Adam: a method for stochastic optimization. In: International conference on learning representations

  28. Lin TY, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for dense object detection. In: International conference on computer vision

  29. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft COCO: common objects in context. In: European conference on computer vision, pp 740–755. Springer

  30. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg AC (2016) SSD: single shot multibox detector. In: European conference on computer vision, pp 21–37. Springer

  31. Ma CY, Chen MH, Kira Z, AlRegib G (2017) TS-LSTM and temporal-inception: exploiting spatiotemporal dynamics for activity recognition. arXiv preprint arXiv:1703.10667

  32. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V et al (2011) Scikit-learn: Machine Learning in Python. J Mach Learn Res 12(Oct):2825–2830

    MathSciNet  MATH  Google Scholar 

  33. Redmon J, Farhadi A (2017) YOLO9000: better, faster, stronger. In: IEEE conference on computer vision and pattern recognition

  34. Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99

  35. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252

    Article  MathSciNet  Google Scholar 

  36. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations

  37. Sutskever I, Martens J, Dahl G, Hinton G (2013) On the importance of initialization and momentum in deep learning. In: International conference on machine learning, pp 1139–1147

  38. Vollmer M, Möllmann KP (2017) Infrared thermal imaging: fundamentals, research and applications. Wiley, New York

    Book  Google Scholar 

  39. Wu B, Iandola F, Jin PH, Keutzer K (2017) SqueezeDet: unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving. In: IEEE conference on computer vision and pattern recognition workshops

  40. Wu Z, Yao T, Fu Y, Jiang Y-G (2017) Deep learning for video classification and captioning. In: Chang S-F (ed) Frontiers of multimedia research. Association for Computing Machinery and Morgan & Claypool, New York, pp 3–29. https://doi.org/10.1145/3122865.3122867

    Chapter  Google Scholar 

  41. Xu H, Das A, Saenko K (2017) R-C3D: region convolutional 3D network for temporal activity detection. In: International conference on computer vision

  42. Zhuang J, Liu Q (2016) Transferred IR pedestrian detector toward distinct scenarios adaptation. Neural Comput Appl 27(3):557–569

    Article  Google Scholar 

Download references

Acknowledgements

This research has been conducted as part of a joint research project with the National Environmental Agency (NEA) of Singapore, sponsored by the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore; EEE Seed Grant for Smart Nation Project, M4081921.040.

Author information

Authors and Affiliations

  1. School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, 639798, Singapore

    Muhammet Bastan, Kim-Hui Yap & Lap-Pui Chau

Authors
  1. Muhammet Bastan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kim-Hui Yap
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lap-Pui Chau
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammet Bastan.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bastan, M., Yap, KH. & Chau, LP. Remote detection of idling cars using infrared imaging and deep networks. Neural Comput & Applic 32, 3047–3057 (2020). https://doi.org/10.1007/s00521-019-04077-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04077-0

Keywords

Search

Navigation