Skip to main content
SpringerLink
Log in

Statistically correlated multi-task learning for autonomous driving

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Autonomous driving research is an emerging area in the machine learning domain. Most existing methods perform single-task learning, while multi-task learning (MTL) is more efficient due to the leverage of shared information between different tasks. However, MTL is challenging because different tasks may have different significance and varying ranges. In this work, we propose an end-to-end deep learning architecture for statistically correlated MTL using a single input image. Statistical correlation of the tasks is handled by including shared layers in the architecture. Later network separates into different branches to handle the difference in the behavior of each task. Training a multi-task model with varying ranges may converge the objective function only with larger values. To this end, we explore different normalization schemes and empirically observe that the inverse validation-loss weighted scheme has best performed. In addition to estimating steering angle, braking, and acceleration, we also estimate the number of lanes on the left and the right side of the vehicle. To the best of our knowledge, we are the first to propose an end-to-end deep learning architecture to estimate this type of lane information. The proposed approach is evaluated on four publicly available datasets including Comma.ai, Udacity, Berkeley Deep Drive, and Sully Chen. We also propose a synthetic dataset GTA-V for autonomous driving research. Our experiments demonstrate the superior performance of the proposed approach compared to the current state-of-the-art methods. The GTA-V dataset and the lane annotations on the four existing datasets will be made publicly available via https://cvlab.lums.edu.pk/scmtl/.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Abbas W, Taj M (2019) Adaptively weighted multi-task learning using inverse validation loss. In: IEEE International conference on acoustics, speech and signal processing

  2. Alamri A, Gumaei A, Al-Rakhami M, Hassan MM, Alhussein M, Fortino G (2020) An effective bio-signal-based driver behavior monitoring system using a generalized deep learning approach. IEEE Access 8:135037–135049

    Article  Google Scholar 

  3. Al-Qizwini M, Barjasteh I, Al-Qassab H, Radha H (2017) Deep learning algorithm for autonomous driving using GoogLeNet. In: IEEE intelligent vehicles symposium. pp 89–96

  4. Bojarski M, Del Testa D, Dworakowski D et al (2016) End to end learning for self-driving cars. arXiv:1604.07316

  5. Bojarski M, Yeres P, Choromanska A, Choromanski K, Firner B, Jackel LD, Muller U (2017) Explaining how a deep neural network trained with end-to-end learning steers a car. arXiv:1704.07911

  6. Campbell M, Egerstedt M, How JP, Murray RM (2010) Autonomous driving in urban environments: approaches, lessons and challenges. Philos Trans Ser A Math Phys Eng Sci 368(1928):4649–72

    Google Scholar 

  7. Chen CA, Seff AK, Xiao J (2015) Deepdriving: learning affordance for direct perception in autonomous driving. In: IEEE international conference on computer vision

  8. Chen Y, Zhao D, Lv L, Zhang Q (2018) Multi-task learning for dangerous object detection in autonomous driving. Inf Sci 432:559–571

    Article  Google Scholar 

  9. Cheng G, Yang C, Yao X, Guo L, Han J (2018) When deep learning meets metric learning: remote sensing image scene classification via learning discriminative cnns. In: IEEE transactions on geoscience and remote sensing

  10. Chen S. https://goo.gl/5wq7fe. Last accessed 07 Dec 2018

  11. Chen Z, Huang X (2017) End-to-end learning for lane keeping of self-driving cars. In: IEEE intelligent vehicles symposium. pp 1856–1860

  12. Chi L, Mu Y (2017) Deep steering: learning end-to-end driving model from spatial and temporal visual cues. arXiv:1708.03798

  13. Chi L, Mu Y (2017) Learning end-to-end autonomous steering model from spatial and temporal visual cues. In: Workshop on visual analysis in smart and connected communities. pp 9–16

  14. Daniel NJH, Lee TD, Liu SC (2017) Delta networks for optimized recurrent network computation. In: International conference on machine learning

  15. Du S, Guo H, Simpson A (2017) Self-driving car steering angle prediction based on image recognition. Technical report, Stanford, CA, USA CS231 course project

  16. Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? The KITTI vision benchmark suite. In: IEEE conference on computer vision and pattern recognition. pp 3354–3361

  17. Hassan MM, Ullah S, Hossain MS et al (2020) An end-to-end deep learning model for human activity recognition from highly sparse body sensor data in internet of medical things environment. J Supercomput

  18. He K, Wang Z, Fu Y, Feng R, Jiang YG, Xue X (2017) Adaptively weighted multi-task deep network for person attribute classification. In: ACM international conference on multimedia. pp 1636–1644

  19. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition. pp 770–778

  20. Hou Y, Ma Z, Liu C, Loy CC (2019) Learning to steer by mimicking features from heterogeneous auxiliary networks. Proc AAAI Conf Artif Intell 33:8433–8440

    Google Scholar 

  21. Innocenti C, Lindén H, Panahandeh G, Svensson L, Mohammadiha N (2017) Imitation learning for vision-based lane keeping assistance. In: Intelligent transportation systems. pp 425–430

  22. Jiang J, Astolfi A (2017) A lateral control assistant for the dynamic model of vehicles subject to state constraints. In: IEEE conference on decision and control. pp 244–249

  23. John V, Mita S, Tehrani H, Ishimaru K (2017) Automated driving by monocular camera using deep mixture of experts. In: IEEE intelligent vehicles symposium. pp 127–134

  24. Kim J, Canny JF (2017) Interpretable learning for self-driving cars by visualizing causal attention. In: IEEE international conference on computer vision. pp 2961–2969

  25. Kingma DP, Ba JL (2014) Adam: a method for stochastic optimization In: Proceedings of the 3rd international conference on learning representations

  26. LeCun Y, Muller U, Ben J, Cosatto E, Flepp B (2005) Off-road obstacle avoidance through end-to-end learning. In: Advances in neural information processing systems. pp 739–746

  27. Liang J, Liu Z, Zhou J, Jiang X, Zhang C, Wang F (2018) Model-protected multi-task learning. arXiv:1809.06546

  28. Lio MD, Mazzalai A, Gurney K, Saroldi A (2018) Biologically guided driver modeling: the stop behavior of human car drivers. IEEE Trans. Int. Trans. Syst. 19(8):2454–2469

    Article  Google Scholar 

  29. Markatopoulou F, Mezaris V, Patras I (2018) Implicit and explicit concept relations in deep neural networks for multi-label video/image annotation. In: IEEE transactions on circuits and systems for video technology

  30. Menéndez-Romero C, Winkler F, Dornhege C, Burgard W (2017) Maneuver planning for highly automated vehicles. In: IEEE intelligent vehicles symposium. pp 1458–1464

  31. Olabiyi O, Martinson E, Chintalapudi V, Guo R (2017) Driver action prediction using deep (bidirectional) recurrent neural network. arXiv:1706.02257

  32. Roberts B, Kaltwang S, Samangooei S, Pender-Bare M, Tertikas K, Redford J (2018) A dataset for lane instance segmentation in urban environments. In: European conference on computer vision

  33. Ruder S (2017) An overview of multi-task learning in deep neural networks. arXiv:1706.05098

  34. Santana E, Hotz G (2016) Learning a driving simulator. arXiv:1608.01230

  35. Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. AAAI Conf Artif Intell 4:12

    Google Scholar 

  36. Taj M, Abbas W (2019) Multi-task learning for autonomous driving, AI for emerging verticals: human-robot computing, sensing and networking, Chapter 13, eds. Shakir, M.Z., Ramza, M.N., IET

  37. Teichmann M, Weber M, Zöllner JM, Cipolla R, Urtasun R (2018) Multinet: real-time joint semantic reasoning for autonomous driving. In: IEEE intelligent vehicles symposium. pp 1013–1020

  38. Udacity: arXiv:1604.073160. Last accessed: 07 Dec 2018

  39. Wang R, Li M, Peng L, Hu Y, Hassan MM, Alelaiwi A (2020) Cognitive multi-agent empowering mobile edge computing for resource caching and collaboration. In: Future generation computer systems. pp 66–74

  40. Wu Y, Chen Z, Liu R, Li F (2018) Lane departure avoidance control for electric vehicle using torque allocation. Math Prob Eng 2018

  41. Xu H, Gao Y, Yu F, Darrell T (2017) End-to-end learning of driving models from large-scale video datasets. In: IEEE conference on computer vision and pattern recognition. pp 3530–3538

  42. Xu H, Gao Y, Yu F, Darrell T (2017) End-to-end learning of driving models from large-scale video datasets. In: IEEE Conference on Computer Vision and Pattern Recognition. pp 3530–3538

  43. Yang J, You X, Wu G, Hassan MM, Almogren A, Guna J (2019) Application of reinforcement learning in UAV cluster task scheduling, In: Future generation computer systems. pp 140–148

  44. Yang Z, Zhang Y, Yu J, Cai J, Luo J (2018) End-to-end multi-modal multi-task vehicle control for self-driving cars with visual perception. arXiv:1604.073161

  45. Yuan C, Hu W, Tian G, Yang S, Wang H (2013) Multi-task sparse learning with beta process prior for action recognition. In: IEEE conference on computer vision and pattern recognition. pp 423–429

  46. Zeng T, Ji S (2015) Deep convolutional neural networks for multi-instance multi-task learning. In: IEEE ICDM. pp 579–588

  47. Zhang Y, Yang Q (2017) A survey on multi-task learning. arXiv:1604.073162

  48. Zhang Y, Yeung DY (2010) A convex formulation for learning task relationships in multi-task learning. In: Proceedings of the conference on uncertainty in artificial intelligence. pp 733–742

  49. Zhou Q, Wang G, Jia K, Zhao Q (2013) Learning to share latent tasks for action recognition. In: IEEE international conference on computer vision. pp 2264–2271

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Lahore University of Management Sciences, Lahore, Pakistan

    Waseem Abbas, Muhammad Fakhir Khan & Murtaza Taj

  2. Department of Computer Science, Information Technology University, Lahore, Pakistan

    Arif Mahmood

Authors
  1. Waseem Abbas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Muhammad Fakhir Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Murtaza Taj
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Arif Mahmood
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Murtaza Taj.

Ethics declarations

Conflict of interest

W. Abbas has now joined Cloud Application Solutions Division, Mentor, A Siemens Business and M. F. Khan is now working at SlashNext Inc. M. Taj is also an adjunct faculty at Ontario Tech University, Canada.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Abbas, W., Khan, M.F., Taj, M. et al. Statistically correlated multi-task learning for autonomous driving. Neural Comput & Applic 33, 12921–12938 (2021). https://doi.org/10.1007/s00521-021-05941-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-05941-8

Keywords

Search

Navigation