Skip to main content
SpringerLink
Log in

Object Detection with Deep Neural Networks for Reinforcement Learning in the Task of Autonomous Vehicles Path Planning at the Intersection

  • Published:
Optical Memory and Neural Networks Aims and scope Submit manuscript

Abstract

Among a number of problems in the behavior planning of an unmanned vehicle the central one is movement in difficult areas. In particular, such areas are intersections at which direct interaction with other road agents takes place. In our work, we offer a new approach to train of the intelligent agent that simulates the behavior of an unmanned vehicle, based on the integration of reinforcement learning and computer vision. Using full visual information about the road intersection obtained from aerial photographs, it is studied automatic detection the relative positions of all road agents with various architectures of deep neural networks (YOLOv3, Faster R-CNN, RetinaNet, Cascade R-CNN, Mask R-CNN, Cascade Mask R-CNN). The possibilities of estimation of the vehicle orientation angle based on a convolutional neural network are also investigated. Obtained additional features are used in the modern effective reinforcement learning methods of Soft Actor Critic and Rainbow, which allows to accelerate the convergence of its learning process. To demonstrate the operation of the developed system, an intersection simulator was developed, at which a number of model experiments were carried out.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.
Fig. 9.
Fig. 10.
Fig. 11.
Fig. 12.
Fig. 13.

Similar content being viewed by others

REFERENCES

  1. Sakib, N., Yao, H., and Zhang, H., Reinforcing classical planning for adversary driving scenarios, arXiv: 1903.08606v1, 2019.

  2. Xu, Z., Tang, C., and Tomizuka, M., Zero-shot deep reinforcement learning driving policy transfer for autonomous vehicles based on robust control, IEEE International Conference on Intelligent Transportation Systems (ITSC), 2019. https://doi.org/10.1109/ITSC.2018.8569612

  3. Kim, M., Path planning and following for autonomous vehicles and its application to intersection with moving obstacles, MSc Thesis, University of Florida, 2017. https://ufdc.ufl.edu/UFE0051086/00001.

  4. Paxton, C., Raman, V., Hager, G.D., and Kobilarov, M., Combining neural networks and tree search for task and motion planning in challenging environments, arXiv:1703.07887v1, 2017.

  5. Tram, T., Jansson, A., Gronberg, R., Sjoberg, J., and Ali, M., Learning negotiating behavior between cars in intersections using deep Q-learning, arXiv:1810.10469v1, 2018.

  6. Jaeyoung, L., Balakrishnan, A., Gaurav, A., Czarnecki, K., and Sedwards, S., WiseMove: A framework for safe deep reinforcement learning for autonomous driving, arXiv:1902.04118, 2019.

  7. Receveur, J.-B., Victor, S., and Melchior, P., Trajectory optimization for autonomous vehicles on crossroads with mobile obstacles, IEEE Intelligent Vehicles Symposium (IV), 2018. https://doi.org/10.1109/IVS.2018.8500379

  8. Chaplot, D.S., Gupta, S., Gupta, A., and Salakhutdinov, R., Modular visual navigation using active neural mapping, 2019. http://www.cs.cmu.edu/~dchaplot/papers/active_neural_mapping.pdf.

  9. Bacchiani, G., Molinari, D., and Patander M., Microscopic traffic simulation by cooperative multi-agent deep reinforcement learning, arXiv:1903.01365v1, 2019.

  10. Computer Vision Annotation Tool (CVAT). https://github.com/opencv/cvat.

  11. Structural Analysis and Shape Descriptors. MinAreaRect. https://docs.opencv.org/3.1.0/d3/dc0/group__imgproc__shape.html.

  12. Fitzgibbon, A.W. and Fisher, R.B., A buyer’s guide to conic fitting, Proc. 5th British Machine Vision Conference, Birmingham, 1995, pp. 513–522.

  13. Shetty, R., Fritz, M., and Schiele, B., Adversarial scene editing: Automatic object removal from weak supervision, arXiv:1806.01911, 2018.

  14. Liu, G., Reda, F. A., Shih, K. J., Wang, T.-C., Tao, A., and Catanzaro, B., Image inpainting for irregular holes using partial convolutions, arXiv:1804.07723, 2018.

    Google Scholar 

  15. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W., OpenAI Gym, arXiv:1606.01540, 2016.

  16. Intersection simulator, MIPT, 2019. https://github.com/cds-mipt/raai-summer-school-2019.

  17. Yudin, D. and Slavioglo, D., Usage of fully convolutional network with clustering for traffic light detection, 7th Mediterranean Conference on Embedded Computing, MECO’2018, 2018, pp. 242–247.

  18. Ester, M., Kriegel, H.P., Sander, J., and Xu, X., A density-based algorithm for discovering clusters in large spatial databases with noise, Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining, Portland, OR, 1996, pp. 226–231.

  19. Redmon, J. and Farhadi, A., YOLOv3: An incremental improvement, arXiv:1804.027672018, 2018.

  20. Ren, S., He, K., Girshick, R., and Sun, J., Faster R-CNN: Towards real-time object detection with region proposal networks, NIPS, 2015.

    Google Scholar 

  21. Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P., Focal loss for dense object detection, arXiv:1 708.02002v2, 2017.

  22. Cai, Z. and Vasconcelos, N., Cascade R-CNN: Delving into high quality object detection, CVPR, 2018.

    Google Scholar 

  23. He, K., Gkioxari, G., Dollár, P., and Girshick, R., Mask R-CNN, arXiv:1703.06870, 2017.

  24. Cai, Z., and Vasconcelos, N., Cascade R-CNN: High quality object detection and instance segmentation, arXiv: 1906.09756, 2019.

  25. Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., Liu, W., and Xiao, B., Deep high-resolution representation learning for visual recognition, arXiv:1908.07919v1, 2019.

  26. Shikunov, M., and Panov, A.I., Hierarchical reinforcement learning approach for the road intersection task, Biologically Inspired Cognitive Architectures 2019, BICA 2019,Advances in Intelligent Systems and Computing, 2020, pp. 495–506.

    Google Scholar 

  27. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S., Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, 2018.

  28. Hessel, M., Modayil, J., van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D., Rainbow: Combining improvements in deep reinforcement learning, arXiv:1710.02298, 2017.

  29. Ziebart, B.D., Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy, Carnegie Mellon University, 2010.

    Google Scholar 

Download references

Funding

This work was partially supported by the Russian Science Foundation, project no. 18-71-00143 (chapters 5 and 6, behavior control with reinforcement learning and experimental results) and by Government of the Russian Federation, agreement no. 075-02-2019-967 (chapters 3 and 4, detection and recognition with computer vision).

Author information

Authors and Affiliations

  1. Moscow Institute of Physics and Technology (National Research University), 141701, Moscow, Russia

    D. A. Yudin, A. Krishtopik, I. Belkin & A. I. Panov

  2. Artificial Intelligence Research Institute, Federal Research Center Computer Science and Control, Russian Academy of Sciences, 119333, Moscow, Russia

    A. Skrynnik & A. I. Panov

Authors
  1. D. A. Yudin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. Skrynnik
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. A. Krishtopik
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. I. Belkin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. A. I. Panov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to D. A. Yudin or A. I. Panov.

Ethics declarations

The authors declare that they have no conflicts of interest.

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yudin, D.A., Skrynnik, A., Krishtopik, A. et al. Object Detection with Deep Neural Networks for Reinforcement Learning in the Task of Autonomous Vehicles Path Planning at the Intersection. Opt. Mem. Neural Networks 28, 283–295 (2019). https://doi.org/10.3103/S1060992X19040118

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.3103/S1060992X19040118

Keywords:

Search

Navigation