Skip to main content

Deep RL Reward Function Design for Lane-Free Autonomous Driving

Part of the Lecture Notes in Computer Science book series (LNAI,volume 13616)


In this paper we present an application of Deep Reinforcement Learning to lane-free traffic, where vehicles do not adhere to the notion of lanes, but are rather able to be located at any lateral position within the road boundaries. This constitutes an entirely different problem domain for autonomous driving compared to lane-based traffic, as vehicles consider the actual two dimensional space available, and their decision making needs to adapt to this concept. We also consider that each vehicle wishes to maintain a (different) desired speed, therefore creating many situations where vehicles need to perform overtaking, and react appropriately to the behaviour of others. As such, in this work, we design a Reinforcement Learning agent for the problem at hand, considering different components of reward functions tied to the environment at various levels of information. Finally, we examine the effectiveness of our approach using the Deep Deterministic Policy Gradient algorithm.


  • Deep reinforcement learning
  • Lane-free traffic
  • Autonomous driving

The research leading to these results has received funding from the European Research Council under the European Union’s Horizon 2020 Research and Innovation programme/ ERC Grant Agreement n. [833915], project TrafficFluid.

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-031-18192-4_21
  • Chapter length: 13 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   69.99
Price excludes VAT (USA)
  • ISBN: 978-3-031-18192-4
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   89.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.


  1. 1.

    Videos showcasing a trained agent with ‘All-Components Reward Function’ can be found at:


  1. Aradi, S.: Survey of deep reinforcement learning for motion planning of autonomous vehicles. IEEE Trans. Intell. Transp. Syst. 23(2), 740–759 (2022)

    CrossRef  Google Scholar 

  2. Badia, A.P., et al.: Agent57: outperforming the Atari human benchmark. In: Proceedings of the 37th International Conference on Machine Learning, PMLR, vol. 119, pp. 507–517 (2020)

    Google Scholar 

  3. Coulom, R.: Efficient selectivity and backup operators in Monte-Carlo tree search. In: van den Herik, H.J., Ciancarini, P., Donkers, H.H.L.M.J. (eds.) CG 2006. LNCS, vol. 4630, pp. 72–83. Springer, Heidelberg (2007).

    CrossRef  Google Scholar 

  4. Di, X., Shi, R.: A survey on autonomous vehicle control in the era of mixed-autonomy: From physics-based to AI-guided driving policy learning. Transp. Res. Part C: Emerg. Technol. 125(4), 103008 (2021)

    Google Scholar 

  5. Gu, S., Lillicrap, T., Sutskever, I., Levine, S.: Continuous deep Q-learning with model-based acceleration. In: Proceedings of The 33rd International Conference on Machine Learning, PMLR, vol. 48, pp. 2829–2838 (2016)

    Google Scholar 

  6. Karafyllis, I., Theodosis, D., Papageorgiou, M.: Two-dimensional cruise control of autonomous vehicles on lane-free roads. In: 60th IEEE conference on Decision and Control (CDC2021), pp. 2683–2689 (2021)

    Google Scholar 

  7. Kendall, A., et al.: Learning to drive in a day. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 8248–8254 (2019)

    Google Scholar 

  8. Kiran, B.R., et al.: Deep reinforcement learning for autonomous driving: a survey. IEEE Trans. Intell. Transp. Syst. PP(99), 1–18 (2021)

    Google Scholar 

  9. Li, C., Czarnecki, K.: Urban driving with multi-objective deep reinforcement learning. In: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 359–367. AAMAS 2019 (2019)

    Google Scholar 

  10. Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. In: Proceedings of the 4th International Conference on Learning Representations, ICLR (2016)

    Google Scholar 

  11. Mnih, V., et al.: Playing atari with deep reinforcement learning (2013)

    Google Scholar 

  12. Papageorgiou, M., Mountakis, K.S., Karafyllis, I., Papamichail, I., Wang, Y.: Lane-free artificial-fluid concept for vehicular traffic. Proc. IEEE 109(2), 114–121 (2021)

    CrossRef  Google Scholar 

  13. Plappert, M.: keras-RL. (2016)

  14. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

  15. Troullinos, D., Chalkiadakis, G., Papamichail, I., Papageorgiou, M.: Collaborative multiagent decision making for lane-free autonomous driving. In: Proceedings of AAMAS-2020, pp. 1335–1343 (2021)

    Google Scholar 

  16. Wu, C., Kreidieh, A.R., Parvate, K., Vinitsky, E., Bayen, A.M.: Flow: a modular learning framework for mixed autonomy traffic. IEEE Trans. Robot. 38, 1–17 (2021)

    Google Scholar 

  17. Yanumula, V.K., Typaldos, P., Troullinos, D., Malekzadeh, M., Papamichail, I., Papageorgiou, M.: Optimal path planning for connected and automated vehicles in lane-free traffic. In: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 3545–3552 (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Athanasia Karalakou .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Karalakou, A., Troullinos, D., Chalkiadakis, G., Papageorgiou, M. (2022). Deep RL Reward Function Design for Lane-Free Autonomous Driving. In: Dignum, F., Mathieu, P., Corchado, J.M., De La Prieta, F. (eds) Advances in Practical Applications of Agents, Multi-Agent Systems, and Complex Systems Simulation. The PAAMS Collection. PAAMS 2022. Lecture Notes in Computer Science(), vol 13616. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-18191-7

  • Online ISBN: 978-3-031-18192-4

  • eBook Packages: Computer ScienceComputer Science (R0)