Visual and Thermal Data for Pedestrian and Cyclist Detection

  • Sarfraz AhmedEmail author
  • M. Nazmul Huda
  • Sujan Rajbhandari
  • Chitta Saha
  • Mark Elshaw
  • Stratis Kanarachos
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11650)


With the continued advancement of autonomous vehicles and their implementation in public roads, accurate detection of vulnerable road users (VRUs) is vital for ensuring safety. To provide higher levels of safety for these VRUs, an effective detection system should be employed that can correctly identify VRUs in all types of environments (e.g. VRU appearance, crowded scenes) and conditions (e.g. fog, rain, night-time). This paper presents optimal methods of sensor fusion for pedestrian and cyclist detection using Deep Neural Networks (DNNs) for higher levels of feature abstraction. Typically, visible sensors have been utilized for this purpose. Recently, thermal sensors system or combination of visual and thermal sensors have been employed for pedestrian detection with advanced detection algorithm. DNNs have provided promising results for improving the accuracy of pedestrian and cyclist detection. This is because they are able to extract features at higher levels than typical hand-crafted detectors. Previous studies have shown that amongst the several sensor fusion techniques that exist, Halfway Fusion has provided the best results in terms of accuracy and robustness. Although sensor fusion and DNN implementation have been used for pedestrian detection, there is considerably less research undertaken for cyclist detection.


Pedestrian detection Cyclist detection Sensor fusion Deep Neural Networks 


  1. 1.
    Angelova, A., Krizhevsky, A., Vanhoucke, V., Ogale, A., Ferguson, D.: Real-time pedestrian detection with deep network cascades. In: Proceedings of the British Machine Vision Conference, pp. 1–12 (2015).
  2. 2.
    Baek, J., Hong, S., Kim, J., Kim, E.: Efficient pedestrian detection at nighttime using a thermal camera. Sens. (Switz.) 17(8), 1850 (2017). Scholar
  3. 3.
    Bertozzi, M., Broggi, A., Caraffi, C., Del Rose, M., Felisa, M., Vezzoni, G.: Pedestrian detection by means of far-infrared stereo vision. Comput. Vis. Image Underst. 106(2–3), 194–204 (2007). Scholar
  4. 4.
    Biswas, S.K., Milanfar, P.: Linear support tensor machine with LSK channels: pedestrian detection in thermal infrared images. IEEE Trans. Image Proc. 26(9), 4229–4242 (2017). Scholar
  5. 5.
    Chang, S.L., Yang, F.T., Wu, W.P., Cho, Y.A., Chen, S.W.: Nighttime pedestrian detection using thermal imaging based on HOG feature. In: Proceedings 2011 International Conference on System Science and Engineering, pp. 694–698. IEEE (2011).
  6. 6.
    Dai, C., Zheng, Y., Li, X.: Layered representation for pedestrian detection and tracking in infrared imagery. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005) - Workshops. vol. 3, pp. 13–13. IEEE (2005).
  7. 7.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 1, pp. 886–893. IEEE (2005).
  8. 8.
    Davis, J., Sharma, V.: Robust detection of people in thermal imagery. In: 2004 Proceedings of the 17th International Conference on Pattern Recognition, ICPR, pp. 713–716. IEEE (2004).
  9. 9.
    European Road Safety Observatory: Traffic Safety Basic Facts 2012. Technical report, European Road Safety Observatory (2012)Google Scholar
  10. 10.
    Gandhi, T., Trivedi, M.M.: Pedestrian protection systems: issues, survey, and challenges. IEEE Trans. Intell. Transp. Syst. 8(3), 413–430 (2007). Scholar
  11. 11.
    Gerónimo, D., López, A.M., Sappa, A.D., Graf, T.: Survey of pedestrian detection for advanced driver assistance systems. IEEE Trans. Pattern Anal. Mach. Intell. 32(7), 1239–1258 (2010). Scholar
  12. 12.
    Gilmore, E.T., Frazier, P.D, Chouikha, M.F.: Improved human detection using image fusion. In: Proceedings of the IEEE ICRA (2009)Google Scholar
  13. 13.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 580–587 (2014)Google Scholar
  14. 14.
    González, A., et al.: Pedestrian detection at day/night time with visible and FIR cameras: a comparison. Sens. (Switz.) 16(6), 1–11 (2016). Scholar
  15. 15.
    Hurney, P., Jones, E., Waldron, P., Glavin, M., Morgan, F.: Night-time pedestrian classification with histograms of oriented gradients-local binary patterns vectors. IET Intell. Transp. Syst. 9(1), 75–85 (2015). Scholar
  16. 16.
    Hwang, S., Park, J., Kim, N., Choi, Y., So, I.: Multispectral pedestrian detection: benchmark dataset and baseline. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1037–1045 (2015)Google Scholar
  17. 17.
    Zhao, J., Cheung, S.C.S.: Human segmentation by fusing visible-light and thermal imaginary. In: 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops, pp. 1185–1192. IEEE (2009).,
  18. 18.
    Kocic, J., Jovicic, N., Drndarevic, V.: Sensors and sensor fusion in autonomous vehicles. In: 2018 26th Telecommunications Forum (TELFOR), pp. 420–425. IEEE (2018).,
  19. 19.
    Lee, J.H., et al.: Robust pedestrian detection by combining visible and thermal infrared cameras. Sens. (Switz.) 15(5), 10580–10615 (2015). Scholar
  20. 20.
    Li, W., Zheng, D., Zhao, T., Yang, M.: An effective approach to pedestrian detection in thermal imagery. In: 2012 8th International Conference on Natural Computation, pp. 325–329. IEEE (2012).
  21. 21.
    Li, X., et al.: A new benchmark for vision-based cyclist detection. In: Proceedings of IEEE Intelligent Vehicles Symposium, pp. 1028–1033 (2016)Google Scholar
  22. 22.
    Li, X., et al.: A unified framework for concurrent pedestrian and cyclist detection. IEEE Trans. Intell. Transp. Syst. 18(2), 269–281 (2017). Scholar
  23. 23.
    Li, Z., Zhang, J., Wu, Q., Geers, G.: Feature enhancement using gradient salience on thermal image. In: 2010 International Conference on Digital Image Computing: Techniques and Applications, pp. 556–562. IEEE (2010).,
  24. 24.
    Liu, J., Zhang, S., Wang, S., Metaxas, D.N.: Multispectral deep neural networks for pedestrian detection. In: British Machine Vision Conference, pp. 1–13 (2016)Google Scholar
  25. 25.
    Mikolajczyk, K., Schmid, C., Zisserman, A.: Human detection based on a probabilistic assembly of robust part detectors. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3021, pp. 69–82. Springer, Heidelberg (2004). Scholar
  26. 26.
    Neagoe, V.E., Ciotec, A.D., Bărar, A.P.: A concurrent neural network approach to pedestrian detection in thermal imagery. In: 2012 9th International Conference on Communications (COMM), pp. 133–136 (2012).
  27. 27.
    Olmeda, D., Armingol, J.M., de la Escalera, A.: Discrete features for rapid pedestrian detection in infrared images. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3067–3072. IEEE (2012).,
  28. 28.
    O’Malley, R., Jones, E., Glavin, M.: Detection of pedestrians in far-infrared automotive night vision using region-growing and clothing distortion compensation. Infrared Phys. Technol. 53(6), 439–449 (2010). Scholar
  29. 29.
    Tian, W., Lauer, M.: Fast and robust cyclist detection for monocular camera systems. In: International joint Conference on Computer Vision Imaging and Computer Graphics Theory and Applications (VISIGRAPP) (2015)Google Scholar
  30. 30.
    Tian, W., Lauer, M.: Detection and orientation estimation for cyclists by max pooled features. In: Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, SCITEPRESS - Science and Technology Publications, pp. 17–26 (2017).
  31. 31.
    Uijlings, J.R.R., van de Sande, K.E.A., Gevers, T., Smeulders, A.W.M.: Selective search for object recognition. Int. J. Comput. Vision 104(2), 154–171 (2013). Scholar
  32. 32.
    Viola, P., Jones, M.J., Snow, D.: Detecting pedestrians using patterns of motion and appearance. In: Proceedings of the 9th IEEE International Conference on Computer Vision, vol. 1, no. 9, pp. 734–741 (2003).
  33. 33.
    Wagner, J., Fischer, V., Herman, M.: Multispectral pedestrian detection using deep fusion convolutional neural networks. In: European Symposium on Artificial Neural Networks (2016)Google Scholar
  34. 34.
    World Health Organisation: Global Status Report on Road Safety 2015 - Summary (2015)Google Scholar
  35. 35.
    Wu, T.E., Tsai, C.C., Guo, J.I.: LiDAR/camera sensor fusion technology for pedestrian detection. In: 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 1675–1678. IEEE (2017).
  36. 36.
    Xu, D., Ouyang, W., Ricci, E., Wang, X., Sebe, N.: Learning cross-modal deep representations for robust pedestrian detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4236–4244. IEEE (2017).,
  37. 37.
    Xu, F., Liu, X., Fujimura, K.: Pedestrian detection and tracking with night vision. IEEE Trans. Intell. Transp. Syst. 6(1), 63–71 (2005). Scholar
  38. 38.
    Chen, Y., Han, C.: Night-time pedestrian detection by visual-infrared video fusion. In: 2008 7th World Congress on Intelligent Control and Automation, pp. 5079–5084. IEEE (2008).
  39. 39.
    Zhang, S., Benenson, R., Omran, M., Hosang, J., Schiele, B.: How far are we from solving pedestrian detection?. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1259–1267 (2016).
  40. 40.
    Zhao, X., He, Z., Zhang, S., Liang, D.: Robust pedestrian detection in thermal infrared imagery using a shape distribution histogram feature and modified sparse representation classification. Pattern Recogn. 48(6), 1947–1960 (2015). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Sarfraz Ahmed
    • 1
    Email author
  • M. Nazmul Huda
    • 1
  • Sujan Rajbhandari
    • 1
  • Chitta Saha
    • 1
  • Mark Elshaw
    • 1
  • Stratis Kanarachos
    • 2
  1. 1.School of Computing, Electronics and MathematicsCoventry UniversityCoventryUK
  2. 2.School of Mechanical, Aerospace and Automotive EngineeringCoventry UniversityCoventryUK

Personalised recommendations