Abstract
Jaywalking is an abnormal pedestrian behavior which significantly increases the risk of road accidents. Owing to this risk, autonomous driving applications should robustly estimate the jaywalking pedestrians. However, the task of robustly estimating jaywalking is not trivial, especially in the case of visible camera-based estimation. In this work, a two-step hierarchical deep learning formulation using visible and thermal camera is proposed to address these challenges. The two steps are comprised of a deep learning-based scene classifier and two scene-specific semantic segmentation frameworks. The scene classifier classifies the visible-thermal image into legal pedestrian crossing and illegal pedestrian crossing scenes. The two scene-specific segmentation frameworks estimate the normal pedestrians and jaywalking pedestrians. The two segmentation frameworks are individually trained on the legal or illegal crossing scenes. The proposed framework is validated on the FLIR public dataset and compared with baseline algorithms. The experimental results show that the proposed hierarchical strategy reports better accuracy than baseline algorithms in real-time.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
John, V., Guo, C., Mita, S., Kidono, K., Guo, C., Ishimaru, K.: Fast road scene segmentation using deep learning and scene-based models. In: ICPR (2016)
John, V., Liu, Z., Guo, C., Mita, S., Kidono, K.: Real-time lane estimation using deep features and extra trees regression. In: Bräunl, T., McCane, B., Rivera, M., Yu, X. (eds.) PSIVT 2015. LNCS, vol. 9431, pp. 721–733. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-29451-3_57
John, V., et al.: Sensor fusion of intensity and depth cues using the ChiNet for semantic segmentation of road scenes. In: IEEE Intelligent Vehicles Symposium, pp. 585–590 (2018)
John, V., Mita, S.: RVNet: deep sensor fusion of monocular camera and radar for image-based obstacle detection in challenging environments. In: Lee, C., Su, Z., Sugimoto, A. (eds.) PSIVT 2019. LNCS, vol. 11854, pp. 351–364. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-34879-3_27
Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. 41, 1–58 (2009)
John, V., Tsuchizawa, S., Liu, Z., Mita, S.: Fusion of thermal and visible cameras for the application of pedestrian detection. Signal Image Video Process. 11, 517–524 (2017)
Dollar, P., Appel, R., Belongie, S., Perona, P.: Fast feature pyramids for object detection. IEEE Trans. Pattern Anal. Mach. Intell. 36, 1532–1545 (2014)
Zhang, S., Benenson, R., Schiele, B.: Filtered channel features for pedestrian detection. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1751–1760 (2015)
Ohn-Bar, E., Trivedi, M.M.: To boost or not to boost? On the limits of boosted trees for object detection. CoRR abs/1701.01692 (2017)
Brazil, G., Yin, X., Liu, X.: Illuminating pedestrians via simultaneous detection & segmentation. CoRR abs/1706.08564 (2017)
Du, X., El-Khamy, M., Lee, J., Davis, L.S.: Fused DNN: a deep neural network fusion approach to fast and robust pedestrian detection. CoRR abs/1610.03466 (2016)
Carvalho, J.F., Vejdemo-Johansson, M., Pokorny, F.T., Kragic, D.: Long-term prediction of motion trajectories using path homology clusters. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019, Macau, SAR, China, November 3–8, 2019, pp. 765–772. IEEE (2019)
Yoo, Y., Yun, K., Yun, S., Hong, J., Jeong, H., Choi, J.: Visual path prediction in complex scenes with crowded moving objects. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2668–2677 (2016)
Javan Roshtkhari, M., Levine, M.D.: An on-line, real-time learning method for detecting anomalies in videos using spatio-temporal compositions. Comput. Vis. Image Underst. 117, 1436–1452 (2013)
Rudenko, A., Palmieri, L., Arras, K.O.: Joint long-term prediction of human motion using a planning-based social force approach. In: 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21–25, 2018, pp. 1–7. IEEE (2018)
Solaimanpour, S., Doshi, P.: A layered HMM for predicting motion of a leader in multi-robot settings. In: ICRA, pp. 788–793. IEEE (2017)
Bera, A., Kim, S., Manocha, D.: Realtime anomaly detection using trajectory-level crowd behavior learning. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1289–1296 (2016)
Chang, M.F., et al.: Argoverse: 3D tracking and forecasting with rich maps (2019)
Medel, J.R., Savakis, A.E.: Anomaly detection in video using predictive convolutional long short-term memory networks. CoRR abs/1612.00390 (2016)
Wu, X., Zhao, W., Yuan, S.: Skeleton-based pedestrian abnormal behavior detection with spatio-temporal model in public places. In: Journal of Physics Conference Series, vol. 1518, p. 012018 (2020)
Hazirbas, C., Ma, L., Domokos, C., Cremers, D.: FuseNet: incorporating depth into semantic segmentation via fusion-based CNN architecture. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10111, pp. 213–228. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54181-5_14
Xu, D., Ricci, E., Yan, Y., Song, J., Sebe, N.: Learning deep representations of appearance and motion for anomalous event detection. CoRR abs/1510.01553 (2015)
Ha, Q., Watanabe, K., Karasawa, T., Ushiku, Y., Harada, T.: MFNet: towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5108–5115 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
John, V., Boyali, A., Thompson, S., Lakshmanan, A., Mita, S. (2021). Visible and Thermal Camera-Based Jaywalking Estimation Using a Hierarchical Deep Learning Framework. In: Sato, I., Han, B. (eds) Computer Vision – ACCV 2020 Workshops. ACCV 2020. Lecture Notes in Computer Science(), vol 12628. Springer, Cham. https://doi.org/10.1007/978-3-030-69756-3_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-69756-3_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-69755-6
Online ISBN: 978-3-030-69756-3
eBook Packages: Computer ScienceComputer Science (R0)