Multimedia Tools and Applications

, Volume 78, Issue 12, pp 15861–15885 | Cite as

A new thermal infrared and visible spectrum images-based pedestrian detection system

  • Redouan LahmyedEmail author
  • Mohamed El Ansari
  • Ayoub Ellahyani


In this paper, we propose a hybrid system for pedestrian detection, in which both thermal and visible images of the same scene are used. The proposed method is achieved in two basic steps: (1) Hypotheses generation (HG) where the locations of possible pedestrians in an image are determined and (2) hypotheses verification (HV), where tests are done to check the presence of pedestrians in the generated hypotheses. HG step segments the thermal image using a modified version of OTSU thresholding technique. The segmentation results are mapped into the corresponding visible image to obtain the regions of interests (possible pedestrians). A post-processing is done on the resulting regions of interests to keep only significant ones. HV is performed using random forest as classifier and a color-based histogram of oriented gradients (HOG) together with the histograms of oriented optical flow (HOOF) as features. The proposed approach has been tested on OSU Color-Thermal, INO Video Analytics and LITIV data sets and the results justify its effectiveness.


Pedestrian detection Thermal images Visible images Random forests Support vector machines (SVMs) Histogram of oriented gradients (HOG) Histograms of oriented optical flow (HOOF) Local binary pattern (LBP) 



  1. 1.
    Akhloufi MA, Porcher C, Bendada A (2013) Fusion of thermal infrared and visible spectrum images for robust pedestrian tracking. In: Proceedings of SPIE, volume, 9076Google Scholar
  2. 2.
    Ansari M El, Lahmyed R, Tremeau A (2018) A hybrid pedestrian detection system based on visible images and lidar data. In: Proceedings of the 13th international joint conference on computer vision, imaging and computer graphics theory and applications - volume 5: VISAPP, pages 325–334. INSTICC, SciTePressGoogle Scholar
  3. 3.
    Bay H, Ess A, Tuytelaars T, Gool LV (2008) Speeded-up robust features (surf). Comput Vis Image Underst 110(3):346–359CrossRefGoogle Scholar
  4. 4.
    Breiman L (2001) Random forests. Mach Learn 45(1):5–32zbMATHCrossRefGoogle Scholar
  5. 5.
    Castillo JC, Serrano-Cuerda J, Sokolova MV, Costa A, Novais P (2012) Multispectrum video for proactive response in intelligent environments. In: 2012 8th international conference on intelligent environments (IE). IEEE, pp 178–185Google Scholar
  6. 6.
    Charfi S, Ansari M El (2018) Computer-aided diagnosis system for colon abnormalities detection in wireless capsule endoscopy images. Multimedia Tools and Applications 77(3):4047–4064CrossRefGoogle Scholar
  7. 7.
    Choi S, Kim E, Lee K, Oh S (2017) Real-time nonparametric reactive navigation of mobile robots in dynamic environments. Robot Auton Syst 91:11–24CrossRefGoogle Scholar
  8. 8.
    Christianini N, Shawe-Taylor J (2000) An introduction to support vector machines and other kernel-based learning methods. Cambridge University Press, United KingdomCrossRefGoogle Scholar
  9. 9.
    Cuntoor N, Kale A, Chellappa R (2003) Combining multiple evidences for gait recognition. In: 2003 international conference on multimedia and Expo, 2003. ICME’03. Proceedings, vol 3. IEEE, pp III–113Google Scholar
  10. 10.
    Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE computer society conference on computer vision and pattern recognition, 2005. CVPR 2005, volume 1. IEEE, pp 886–893Google Scholar
  11. 11.
    Davis JW, Sharma V (2007) Background-subtraction using contour-based fusion of thermal and visible imagery. Comput Vis Image Underst 106(2):162–182CrossRefGoogle Scholar
  12. 12.
    Dollár P, Appel R, Belongie S, Perona P (2014) Fast feature pyramids for object detection. IEEE Trans Pattern Anal Mach Intell 36(8):1532–1545CrossRefGoogle Scholar
  13. 13.
    Elguebaly T, Bouguila N (2011) A nonparametric bayesian approach for enhanced pedestrian detection and foreground segmentation. In: 2011 IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 21–26Google Scholar
  14. 14.
    Elguebaly T, Bouguila N (2013) Finite asymmetric generalized gaussian mixture models learning for infrared object detection. Comput Vis Image Underst 117(12):1659–1671CrossRefGoogle Scholar
  15. 15.
    Ellahyani A, Ansari M El, Jaafari I El (2016) Traffic sign detection and recognition based on random forests. Appl Soft Comput 46:805–815CrossRefGoogle Scholar
  16. 16.
    Fendri E, Boukhriss RR, Hammami M (2017) Fusion of thermal infrared and visible spectra for robust moving object detection. Pattern Anal Applic 20(4):907–926MathSciNetCrossRefGoogle Scholar
  17. 17.
    Foster JP, Nixon MS, Prügel-Bennett A (2003) Automatic gait recognition using area-based metrics. Pattern Recogn Lett 24(14):2489–2497CrossRefGoogle Scholar
  18. 18.
    Gascuena JM, Serrano-Cuerda J, Castillo JC, Fernández-Caballero A, López MT (2014) A multi-agent system for infrared and color video fusion. In: Trends in practical applications of heterogeneous multi-agent systems. The PAAMS collection. Springer, pp 131–138Google Scholar
  19. 19.
    Gavrila DM, Munder S (2007) Multi-cue pedestrian detection and tracking from a moving vehicle. Int J Comput Vis 73(1):41–59CrossRefGoogle Scholar
  20. 20.
    Ge J, Luo Y, Tei G (2009) Real-time pedestrian detection and tracking at nighttime for driver-assistance systems. IEEE Trans Intell Transp Syst 10(2):283–298CrossRefGoogle Scholar
  21. 21.
    Guo L, Ge P-S, Zhang M-H, Li L-H, Zhao Y-B (2012) Pedestrian detection for intelligent transportation systems combining adaboost algorithm and support vector machine. Expert Systems with Applications 39(4):4274–4286CrossRefGoogle Scholar
  22. 22.
    Herrmann C, Müller T, Willersinn D, Beyerer J (2016) Real-time person detection in low-resolution thermal infrared imagery with mser and cnns. In: SPIE security+ defence, pp 99870I–99870I. International society for optics and photonicsGoogle Scholar
  23. 23.
    Huang D-Y, Wang C-H (2009) Optimal multi-level thresholding using a two-stage otsu optimization approach. Pattern Recogn Lett 30(3):275–284CrossRefGoogle Scholar
  24. 24.
    Hwang S, Park J, Kim N, Choi Y, Kweon IS (2015) Multispectral pedestrian detection benchmark dataset and baseline. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1037–1045Google Scholar
  25. 25.
    Ino, “ino video analytics dataset.” Accessed 6 Sept 2017
  26. 26.
    John V, Mita S, Liu Z, Qi B (2015) Pedestrian detection in thermal images using adaptive fuzzy c-means clustering and convolutional neural networks. In: 2015 14th IAPR international conference on machine vision applications (MVA). IEEE, pp 246–249Google Scholar
  27. 27.
    Jungling K, Arens M (2009) Feature based person detection beyond the visible spectrum. In: IEEE computer society conference on computer vision and pattern recognition workshops, 2009. CVPR Workshops 2009. IEEE, pp 30–37Google Scholar
  28. 28.
    Källhammer J-E, Smith K, Matsangas P (2016) Modeling ratings of in-vehicle alerts to pedestrian by leveraging field operational tests data in a controlled laboratory study. Transportation Research Part F: Traffic Psychology and BehaviourGoogle Scholar
  29. 29.
    Kassani PH, Teoh ABJ (2016) A new sparse model for traffic sign classification using soft histogram of oriented gradients. Appl Soft ComputGoogle Scholar
  30. 30.
    Kim K, Chalidabhongse TH, Harwood D, Davis L (2005) Real-time foreground–background segmentation using codebook model. Real-Time Image 11(3):172–185CrossRefGoogle Scholar
  31. 31.
    Lahmyed R, Ansari M El (2016) Multisensors-based pedestrian detection system. In: 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA). IEEE, pp 1–4Google Scholar
  32. 32.
    Lee L, Dalley G, Tieu K (2003) Learning pedestrian models for silhouette refinement. In: ICCV, vol 1, pp 663–670Google Scholar
  33. 33.
    Level Otsu N (1979) A threshold selection method from gray-level histogram. IEEE Trans Syst Man Cybern B Cybern 9(1):62–66CrossRefGoogle Scholar
  34. 34.
    Li H, Zhu J, Tao D (2018) Asymmetric projection and dictionary learning with listwise and identity consistency constraints for person re-identification. IEEE Access 6:37977–37990CrossRefGoogle Scholar
  35. 35.
    Li Jianfu, Gong Weiguo, Li W, Liu X (2010) Robust pedestrian detection in thermal infrared imagery using the wavelet transform. Infrared Phys Technol 53(4):267–273CrossRefGoogle Scholar
  36. 36.
    Li Z, Bo Wu, Nevatia S (2007) Pedestrian detection in infrared images based on local shape features. In: 2007 IEEE conference on computer vision and pattern recognition CVPR’07. IEEE, pp 1–8Google Scholar
  37. 37.
    Liang C-W, Juang C-F (2015) Moving object classification using local shape and hog features in wavelet-transformed space with hierarchical svm classifiers. Appl Soft Comput 28:483–497CrossRefGoogle Scholar
  38. 38.
    Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110CrossRefGoogle Scholar
  39. 39.
    Lucas BD, Kanade T, et al (1981) An iterative image registration technique with an application to stereo visionGoogle Scholar
  40. 40.
    Morales Y, Miyashita T, Hagita N (2017) Social robotic wheelchair centered on passenger and pedestrian comfort. Robot Auton Syst 87:355–362CrossRefGoogle Scholar
  41. 41.
    Nanda H, Davis L (2002) Probabilistic template based pedestrian detection in infrared videos. In: Intelligent vehicle symposium 2002. IEEE, volume 1, pp 15–20Google Scholar
  42. 42.
    Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern Recogn 29(1):51–59CrossRefGoogle Scholar
  43. 43.
    Ouloul IM, Moutakki Z, Afdel K, Amghar A (2018) Improvement of age estimation using an efficient wrinkles descriptor. Multimedia Tools and Applications, pp 1–35Google Scholar
  44. 44.
    Perš J, Kristan M, Perše M, Kovačič S (2007) Motion based human identification using histograms of optical flow. naGoogle Scholar
  45. 45.
    Perš J, Sulić V, Kristan M, Perše M, Polanec K, Kovačič S (2010) Histograms of optical flow for efficient representation of body motion. Pattern Recog Lett 31(11):1369–1376CrossRefGoogle Scholar
  46. 46.
    Premebida C, Ludwig O, Nunes U (2009) Lidar and vision-based pedestrian detection system. J Field Rob 26(9):696–711CrossRefGoogle Scholar
  47. 47.
    Qingbo J, Enze Z, Xinqi Y, Yu X, Yun L (2016) Face recognition method based on hog and dmma from single training sample. Multimedia Tools and Applications, 75(21):13163–13177CrossRefGoogle Scholar
  48. 48.
    Radman A, Suandi SA (2018) Robust face pseudo-sketch synthesis and recognition using morphological-arithmetic operations and hog-pca. Multimedia Tools and Applications. pp 1–22Google Scholar
  49. 49.
    San-Biagio M, Crocco M, Cristani M (2012) Recursive segmentation based on higher order statistics in thermal imaging pedestrian detection. In: 2012 5th international symposium on communications control and signal processing (ISCCSP). IEEE, pp 1–4Google Scholar
  50. 50.
    Serrano-Cuerda J (2014) Robust human detection through fusion of color and infrared video. ELCVIA: Electronic Letters On Computer Vision And Image Analysis 13(2):0017–18CrossRefGoogle Scholar
  51. 51.
    Souaidi M, Abdelouahad AA, Ansari M El (2017) A fully automated ulcer detection system for wireless capsule endoscopy images. In: 2017 International Conference on Advanced Technologies for Signal and Image Processing (ATSIP). IEEE, pP 1–6Google Scholar
  52. 52.
    Souaidi M, Abdelouahed AA, Ansari M El (2018) Multi-scale completed local binary patterns for ulcer detection in wireless capsule endoscopy images. Multimedia Tools and Applications, pp 1–18Google Scholar
  53. 53.
    Souaidi M, Charfi S, Abdelouahad AA, Ansari M El (2018) New features for wireless capsule endoscopy polyp detection. In: 2018 international conference on intelligent systems and computer vision (ISCV). IEEE, pp 1–6Google Scholar
  54. 54.
    Sun Hao, Wang Cheng, Wang B, El-Sheimy N (2011) Pyramid binary pattern features for real-time pedestrian detection from infrared videos. Neurocomputing 74 (5):797–804CrossRefGoogle Scholar
  55. 55.
    Tao D, Guo Y, Li Y, Gao X (2018) Tensor rank preserving discriminant analysis for facial recognition. IEEE Trans on Image Process 27(1):325–334MathSciNetzbMATHCrossRefGoogle Scholar
  56. 56.
    Torabi A, Massé G, Bilodeau G-A (2012) An iterative integrated framework for thermal–visible image registration, sensor fusion, and people tracking for video surveillance applications. Comput Vis Image Underst 116(2):210–221CrossRefGoogle Scholar
  57. 57.
    Usher JM, McCool R, Strawderman L, Carruth DW, Bethel CL, May DC (2017) Simulation modeling of pedestrian behavior in the presence of unmanned mobile robots. Simul Model Pract Theory 75:96–112CrossRefGoogle Scholar
  58. 58.
    Vapnik VN, Vapnik V (1998) Statistical learning theory, volume 1. Wiley, New YorkzbMATHGoogle Scholar
  59. 59.
    Wagner J, Fischer V, Herman M, networks SB (2016) Multispectral pedestrian detection using deep fusion convolutional neural. In: 24th European symposium on artificial neural networks computational intelligence and machine learning (ESANN), pp 509–514Google Scholar
  60. 60.
    Wang Y, Wang Z, Tao D, Zhuo S, Xu X, Pu S, Song M (2017) Allfocus: patch-based video out-of-focus blur reconstruction. IEEE Trans Circuits Syst Video Technol 27(9):1895–1908CrossRefGoogle Scholar
  61. 61.
    Yang T, Fu D, Pan S (2017) Pedestrian tracking for infrared image sequence based on trajectory manifold of spatio-temporal slice. Multimedia Tools and Applications 76(8):11021–11035CrossRefGoogle Scholar
  62. 62.
    Zhang J, Li F-W, Nie W-Z, Li W-H, Su Y-T (2016) Visual attribute detction for pedestrian detection. Multimedia Tools and Applications, pp 1–18Google Scholar
  63. 63.
    Zin TT, Takahashi H, Hama H, Toriu T (2011) Fusion of infrared and visible images for robust person detection. INTECH Open Access PublisherGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.LabSIV, Department of Computer Science, Faculty of ScienceIbn Zohr UniversityAgadirMorocco

Personalised recommendations