Skip to main content
SpringerLink
Log in

Road obstacles positional and dynamic features extraction combining object detection, stereo disparity maps and optical flow data

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

One of the most relevant tasks in an intelligent vehicle navigation system is the detection of obstacles. It is important that a visual perception system for navigation purposes identifies obstacles, and it is also important that this system can extract essential information that may influence the vehicle’s behavior, whether it will be generating an alert for a human driver or guide an autonomous vehicle in order to be able to make its driving decisions. In this paper we present an approach for the identification of obstacles and extraction of class, position, depth and motion information from these objects that employs data gained exclusively from passive vision. We use a convolutional neural network for the obstacles detection, optical flow for the analysis of movement of the detected obstacles, both in relation to the direction and in relation to the intensity of the movement, and also stereo vision for the analysis of distance of obstacles in relation to the vehicle. We performed our experiments on two different datasets, and the results obtained showed a good efficacy from the use of depth and motion patterns to assess the obstacles’ potential threat status.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. http://www.cvlibs.net/datasets/kitti/raw_data.php.

  2. http://www.lrm.icmc.usp.br/dataset.

References

  1. Bleyer, M.: VU Stereo Vision. Karlsplatz, Vienna, Austria (2013)

    Google Scholar 

  2. Bouchafa, S., Zavidovique, B.: Obstacle detection ”for free”; in the c-velocity space. In: 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), pp. 308–313 (2011). https://doi.org/10.1109/ITSC.2011.6082872

  3. Chanawangsa, P., Chen, C.W.: A novel video analysis approach for overtaking vehicle detection. In: 2013 International Conference on Connected Vehicles and Expo (ICCVE), pp. 802–807 (2013). https://doi.org/10.1109/ICCVE.2013.6799901

  4. Chen, Y.S., Tsai, A.C., Lin, T.T.: Road environment recognition method in complex traffic situations based on stereo vision. In: 2012 12th International Conference on ITS Telecommunications, pp. 180–184 (2012). https://doi.org/10.1109/ITST.2012.6425161

  5. Commission, I.E.: Safety of laser products. Part 1: equipment classification, requirements and user’s guide (2001)

  6. Deo, N., Rangesh, A., Trivedi, M.M.: How would surround vehicles move? A unified framework for maneuver classification and motion prediction. CoRR abs/1801.06523 (2018). http://arxiv.org/abs/1801.06523

  7. Dosovitskiy, A., Fischer, P., Ilg, E., Husser, P., Hazirbas, C., Golkov, V., v d Smagt, P., Cremers, D., Brox, T.: Flownet: learning optical flow with convolutional networks. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2758–2766 (2015). https://doi.org/10.1109/ICCV.2015.316

  8. Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. In: Bigun, J., Gustavsson, T. (eds.) Image Analysis: 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29–July 2, 2003 Proceedings. Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 363–370 (2003). https://doi.org/10.1007/3-540-45103-X_50

  9. Fernandes, L.C., Souza, J.R., Pessin, G., Shinzato, P.Y., Sales, D., Mendes, C., Prado, M., Klaser, R., Magalhes, A.C., Hata, A., Pigatto, D., Branco, K.C., Osorio, F.S., Wolf, D.F.: Carina intelligent robotic car: architectural design and applications. J. Syst. Archit. 60(4), 372–392 (2014). https://doi.org/10.1016/j.sysarc.2013.12.003

    Article  Google Scholar 

  10. Fleet, D.J., Weiss, Y.: Optical flow estimation (2005)

  11. Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the kitti dataset. Int. J. Rob. Res. 32(11), 1231–1237 (2013). https://doi.org/10.1177/0278364913491297

    Article  Google Scholar 

  12. Giosan, I., Nedevschi, S.: Superpixel-based obstacle segmentation from dense stereo urban traffic scenarios using intensity, depth and optical flow information. In: 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), pp. 1662–1668 (2014). https://doi.org/10.1109/ITSC.2014.6957932

  13. Gupta, K., Upadhyay, S., Gandhi, V., Krishna, K.M.: Small obstacle detection using stereo vision for autonomous ground vehicle. In: Proceedings of the Advances in Robotics, ACM, New York, NY, USA, AIR ’17, pp. 25:1–25:6 (2017). https://doi.org/10.1145/3132446.3134889

  14. He, K., Gkioxari, G., Dollr, P., Girshick, R.: Mask r-cnn. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988 (2017). https://doi.org/10.1109/ICCV.2017.322

  15. Huang, Y., Liu, S.: Multi-class obstacle detection and classification using stereovision and improved active contour models. IET Intell. Transp. Syst. 10(3), 197–205 (2016). https://doi.org/10.1049/iet-its.2014.0308

    Article  Google Scholar 

  16. Hne, C., Heng, L., Lee, G.H., Fraundorfer, F., Furgale, P., Sattler, T., Pollefeys, M.: 3d visual perception for self-driving cars using a multi-camera system: calibration, mapping, localization, and obstacle detection, automotive vision: challenges, trends, technologies and systems for vision-based intelligent vehicles. Image Vis. Comput. 68, 14–27 (2017). https://doi.org/10.1016/j.imavis.2017.07.003

    Article  Google Scholar 

  17. Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: evolution of optical flow estimation with deep networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1647–1655 (2017). https://doi.org/10.1109/CVPR.2017.179

  18. Jawed, S., Boumaiza, E., Grabocka. J., Schmidt-Thieme, L.: Data-driven vehicle trajectory forecasting. CoRR abs/1902.05400 (2019). http://arxiv.org/abs/1902.05400

  19. Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering (2007)

  20. Li, J., Chen, M.: On-road multiple obstacles detection in dynamical background. In: 2014 Sixth International Conference on Intelligent Human-Machine Systems and Cybernetics, vol. 1, pp. 102–105 (2014), https://doi.org/10.1109/IHMSC.2014.33

  21. LiDAR-UKcom.: LIDAR. UK (2015). http://www.lidar-uk.com/

  22. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision - ECCV 2014, pp. 740–755. Springer International Publishing, Cham (2014)

  23. Liu, S., Huang ,Y., Zhang, R.: Obstacle recognition for adas using stereovision and snake models. In: 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), pp. 99–104 (2014). https://doi.org/10.1109/ITSC.2014.6957673

  24. Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proceedings of the 7th International Joint Conference on Artificial Intelligence - Volume 2, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, IJCAI’81, pp. 674–679 (1981). http://dl.acm.org/citation.cfm?id=1623264.1623280

  25. Mitzel, D., Floros, G., Sudowe, P., van der Zander, B., Leibe, B.: Real time vision based multi-person tracking for mobile robotics and intelligent vehicles. In: Jeschke, S., Liu, H., Schilberg, D. (eds.) Intelligent Robotics and Applications, pp. 105–115. Springer, Berlin Heidelberg, Berlin, Heidelberg (2011)

  26. Poddar, A., Ahmed, S.T., Puhan, N.B.: (2015) Adaptive saliency-weighted obstacle detection for the visually challenged. In: 2015 2nd International Conference on Signal Processing and Integrated Networks (SPIN), pp. 477–482. https://doi.org/10.1109/SPIN.2015.7095312

  27. Prabhakar, G., Kailath, B., Natarajan, S., Kumar, R.: (2017) Obstacle detection and classification using deep learning for tracking in high-speed autonomous driving. In: 2017 IEEE Region 10 Symposium (TENSYMP), pp. 1–6. https://doi.org/10.1109/TENCONSpring.2017.8069972

  28. Rateke, T., von Wangenheim, A.: Systematic literature review for passive vision road obstacle detection. Tech. rep., Brazilian Institute for Digital Convergence - INCoD (2018). https://doi.org/10.13140/RG.2.2.10198.14408

  29. Rateke, T., von Wangenheim, A.: Passive vision road obstacle detection: a literature mapping. Int. J. Comput. Appl. (2020). https://doi.org/10.1080/1206212X.2020.1758877

    Article  Google Scholar 

  30. Ren, S., He, K., Girshick , R., Sun, J.: Faster r-cnn: towards real-time object detection with region proposal networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1. MIT Press, Cambridge, MA, USA, NIPS’15, pp. 91–99 (2015). http://dl.acm.org/citation.cfm?id=2969239.2969250

  31. Sengar, S.S., Mukhopadhyay, S.: Motion detection using block based bi-directional optical flow method. J. Vis. Commun. Image Rep. 49(C), 89–103 (2017a)

    Article  Google Scholar 

  32. Sengar, S.S., Mukhopadhyay, S.: Moving object detection based on frame difference and w4. Sig. Image Video Process. 11(7), 1357–1364 (2017b). https://doi.org/10.1007/s11760-017-1093-8

    Article  Google Scholar 

  33. Sengar, S.S., Mukhopadhyay, S.: Motion segmentation-based surveillance video compression using adaptive particle swarm optimization. Neural Comput. Appl. 32(15), 11443–11457 (2020a). https://doi.org/10.1007/s00521-019-04635-6

    Article  Google Scholar 

  34. Sengar, S.S., Mukhopadhyay, S.: Moving object detection using statistical background subtraction in wavelet compressed domain. Multimed. Tools Appl. 79(9), 5919–5940 (2020). https://doi.org/10.1007/s11042-019-08506-z

    Article  Google Scholar 

  35. Shinzato, P.Y., dos Santos, T.C., Rosero, L.A., Ridel, D.A., Massera, C.M., Alencar, F., Batista, M.P., Hata, A.Y., Orio F.S., Wolf, D.F.: Carina dataset: an emerging-country urban scenario benchmark for road detection systems. In: 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), pp. 41–46 (2016). https://doi.org/10.1109/ITSC.2016.7795529

  36. STANDARD AN: American National Standard for Safe use of Lasers Outdoors. Orlando, FL (2005)

  37. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818–2826 (2016). https://doi.org/10.1109/CVPR.2016.308

  38. Urmson, C., Anhalt, J., Bae, H., Bagnell, J.A.D, Baker, C.R., Bittner, R.E., Brown, T., Clark, M.N., Darms, M., Demitrish, D., Dolan, J.M., Duggins, D., Ferguson, D., Galatali, T., Geyer, C.M., Gittleman, M., Harbaugh, S., Hebert, M., Howard, T., Kolski, S., Likhachev, M., Litkouhi, B., Kelly, A., McNaughton, M., Miller, N., Nickolaou, J., Peterson, K., Pilnick, B., Rajkumar, R., Rybski, P., Sadekar, V., Salesky, B., Seo, Y.W., Singh, S., Snider, J.M., Struble, J.C., Stentz, A.T., Taylor, M., Whittaker, W.R.L., Wolkowicki, Z., Zhang, W., Ziglar, J.: Autonomous driving in urban environments: Boss and the urban challenge. Journal of Field Robotics - 2007 DARPA Urban Challenge 25(8), 425–466 (2008)

  39. Yao., Y., Xu, M., Choi, C., Crandall, DJ., Atkins, E.M., Dariush, B.: Egocentric vision-based future vehicle localization for intelligent driving assistance systems. CoRR abs/1809.07408 (2018). http://arxiv.org/abs/1809.07408

Download references

Author information

Authors and Affiliations

  1. Graduate Program in Computer Science (PPGCC), Department of Informatics and Statistics, Federal University of Santa Catarina, Florianópolis, SC, Brazil

    Thiago Rateke & Aldo von Wangenheim

  2. Image Processing and Computer Graphics Lab (LAPIX), National Institute for Digital Convergence (INCoD), Florianópolis, Brazil

    Thiago Rateke & Aldo von Wangenheim

Authors
  1. Thiago Rateke
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aldo von Wangenheim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thiago Rateke.

Ethics declarations

Conflict of interest

The authors declare that there are no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This study was financed in part by the Coordenao de Aperfeioamento de Pessoal de Nvel Superior - Brasil (CAPES) - Finance Code 001. CAPES (Brazilian Federal Agency for Support and Evaluation of Graduate Education). It was also supported by the Brazilian National Institute for Digital Convergence (INCoD), a research unit of the Brazilian National Institutes for Science and Technology Program (INCT) of the Brazilian National Council for Science and Technology (CNPq).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rateke, T., Wangenheim, A.v. Road obstacles positional and dynamic features extraction combining object detection, stereo disparity maps and optical flow data. Machine Vision and Applications 31, 73 (2020). https://doi.org/10.1007/s00138-020-01126-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-020-01126-w

Keywords

Search

Navigation