Skip to main content
SpringerLink
Log in

Accelerating autonomy: an integrated perception digital platform for next generation self-driving cars using faster R-CNN and DeepLabV3

  • Application of soft computing
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

Building robust perception systems for autonomous vehicles requires addressing multimodal data fusion and large-scale deep network training challenges. This work puts forth an integrated methodology based on specialized deep learning architectures including Faster R-CNN and DeepLabV3 adapted for autonomous driving tasks. The approach encompasses a robust data curation pipeline handling multimodal data preprocessing, including sensor synchronization for temporal alignment, normalization using standardization techniques, quality enhancement via noise filtering and controlled augmentation via techniques like cropping and brightness variation. For perception, Faster R-CNN leverages a region proposal network to generate candidate object bounding boxes, followed by a classification network using Feature Pyramid Networks for multi-scale object detection and localization. DeepLabV3 adopts atrous convolutions with varying dilation rates and multiple spatial pyramid pooling modules for semantic segmentation and pixel-level scene understanding. Further elements include object tracking using similarity metrics between extracted features, trajectory forecasting based on past motions, sensor fusion mechanisms like early and mid-level fusion to combine complementary modalities, and coordinated training procedures leveraging joint losses tailored for autonomous driving datasets. Extensive comparative benchmarks demonstrate the proposed framework achieves state-of-the-art performance, with top scores of 94% accuracy, 93% precision and 94% AUC and low error rates such as MSE, RMSE, and MAE on tasks corresponding to object localization, segmentation and scene understanding. Ablation studies provide detailed insights into optimal configurations. The comprehensive methodology unifies deep neural perception pipelines tailored for autonomous driving, serving as a crucial enabler for next-generation intelligent vehicle systems. Thorough experimentation validates its effectiveness in handling diverse sensory data and complex real-world scenarios.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Data availability

Inquiries about data availability should be addressed to the authors.

References

  • Abdar M, Pourpanah F, Hussain S, Rezazadegan D, Liu L, Ghavamzadeh M, Fieguth P, Cao X, Khosravi A, Acharya UR, Makarenkov V (2021) A review of uncertainty quantification in deep learning: techniques, applications and challenges. Inf Fusion 76:243–297

    Article  Google Scholar 

  • Adhikari M, Hazra A, Menon VG, Chaurasia BK, Mumtaz S (2021) A roadmap of next-generation wireless technology for 6G-enabled vehicular networks. IEEE Internet Things Mag 4(4):79–85

    Article  Google Scholar 

  • Ali ES, Hasan MK, Hassan R, Saeed RA, Hassan MB, Islam S, Nafi NS, Bevinakoppa S (2021) Machine learning technologies for secure vehicular communication in internet of vehicles: recent advances and applications. Secur Commun Netw 2021:1–23

    Google Scholar 

  • Ali M, Yin B, Bilal H et al (2023) Advanced efficient strategy for detection of dark objects based on spiking network with multi-box detection. Multimed Tools Appl. https://doi.org/10.1007/s11042-023-16852-2

    Article  Google Scholar 

  • An D, Liu J, Zhang M, Chen X, Chen M, Sun H (2020) Uncertainty modeling and runtime verification for autonomous vehicles driving control: a machine learning-based approach. J Syst Softw 167:110617

    Article  Google Scholar 

  • Aslam S, Muhammad. (2021) L 2–L∞ control for delayed singular markov switch system with nonlinear actuator faults. Int J Fuzzy Syst 23(7):2297–2308

    Article  Google Scholar 

  • Bachute MR, Subhedar JM (2021) Autonomous driving architectures: insights of machine learning and deep learning algorithms. Mach Learn Appl 6:100164

    Google Scholar 

  • Bautista C, Mester G (2023) Internet of things in self-driving cars environment. Interdiscip Descr Complex Syst: INDECS 21(2):188–198

    Article  Google Scholar 

  • Beiker S (2023) Next-generation Sensors for Automated Road Vehicles (No. EPR2023003)

  • Bendiab G, Hameurlaine A, Germanos G, Kolokotronis N, Shiaeles S (2023) Autonomous vehicles security: challenges and solutions using blockchain and artificial intelligence. IEEE Trans Intell Transp Syst 24:3614

    Article  Google Scholar 

  • Butt FA, Chattha JN, Ahmad J, Zia MU, Rizwan M, Naqvi IH (2022) On the integration of enabling wireless technologies and sensor fusion for next-generation connected and autonomous vehicles. IEEE Access 10:14643–14668

    Article  Google Scholar 

  • Chen Z (2019) Observer-based dissipative output feedback control for network T-S fuzzy systems under time delays with mismatch premise. Nonlinear Dyn 95:2923–2941

    Article  Google Scholar 

  • Dai X, Hou J, Li Q, Ullah R, Ni Z, Liu Y (2020) Reliable control design for composite-driven scheme based on delay networked T-S fuzzy system. Int J Robust Nonlinear Control 30(4):1622–1642

    Article  MathSciNet  Google Scholar 

  • Dou H, Liu Y, Chen S et al (2023) A hybrid CEEMD-GMM scheme for enhancing the detection of traffic flow on highways. Soft Comput 27:16373–16388. https://doi.org/10.1007/s00500-023-09164-y

    Article  Google Scholar 

  • Gao L, Zhou R, Zhang K (2023) Scenario generation for autonomous vehicles with deep-learning-based heterogeneous driver models: implementation and verification. Sensors 23(9):4570

    Article  Google Scholar 

  • Grigorescu S, Trasnea B, Cocias T, Macesanu G (2020) A survey of deep learning techniques for autonomous driving. J Field Robot 37(3):362–386

    Article  Google Scholar 

  • Hasanujjaman M, Chowdhury MZ, Jang YM (2023) Sensor fusion in autonomous vehicle with traffic surveillance camera system: detection, localization, and AI networking. Sensors 23(6):3335

    Article  Google Scholar 

  • Haseeb K, Rehman A, Saba T, Bahaj SA, Wang H, Song H (2023) Efficient and trusted autonomous vehicle routing protocol for 6G networks with computational intelligence. ISA Trans 132:61–68

    Article  Google Scholar 

  • Hoffmann JE, Tosso HG, Santos MMD, Justo JF, Malik AW, Rahman AU (2020) Real-time adaptive object detection and tracking for autonomous vehicles. IEEE Trans Intell Veh 6(3):450–459

    Article  Google Scholar 

  • Hu Z, Lou S, Xing Y, Wang X, Cao D, Lv C (2022) Review and perspectives on driver digital twin and its enabling technologies for intelligent vehicles. IEEE Trans Intell Veh. https://doi.org/10.1109/TIV.2022.3195635

    Article  Google Scholar 

  • Jain DK, Dutta AK, Verdú E, Alsubai S, Sait ARW (2023) An automated hyperparameter tuned deep learning model enabled facial emotion recognition for autonomous vehicle drivers. Image vis Comput 133:104659

    Article  Google Scholar 

  • Janai J, Güney F, Behl A, Geiger A (2020) Computer vision for autonomous vehicles: problems, datasets and state of the art. Found Trends Comput Graph vis 12(1–3):1–308

    Article  Google Scholar 

  • Kato S, Tokunaga S, Maruyama Y, Maeda S, Hirabayashi M, Kitsukawa Y, Monrroy A, Ando T, Fujii Y, Azumi T (2018) Autoware on board: Enabling autonomous vehicles with embedded systems. In 2018 ACM/IEEE 9th International Conference on Cyber-Physical Systems (ICCPS) (pp. 287–296). IEEE

  • Kiran BR, Sobh I, Talpaert V, Mannion P, Al Sallab AA, Yogamani S, Pérez P (2021) Deep reinforcement learning for autonomous driving: a survey. IEEE Trans Intell Transp Syst 23(6):4909–4926

    Article  Google Scholar 

  • Le-Khac NA, Jacobs D, Nijhoff J, Bertens K, Choo KKR (2020) Smart vehicle forensics: Challenges and case study. Futur Gener Comput Syst 109:500–510

    Article  Google Scholar 

  • Li Q, Hou J (2021) Fault detection for asynchronous T-S fuzzy networked Markov jump systems with new event-triggered scheme. IET Control Theory Appl 15(11):1461–1473

    Article  MathSciNet  Google Scholar 

  • Li G, Ji Z, Qu X (2022) Stepwise domain adaptation (SDA) for object detection in autonomous vehicles using an adaptive CenterNet. IEEE Trans Intell Transp Syst 23(10):17729–17743

    Article  Google Scholar 

  • Litimein H, Huang ZY, Aslam MS (2023) Circular formation control with collision avoidance based on probabilistic position. Intell Autom Soft Comput 37(1):321

    Article  Google Scholar 

  • Liu L, Zhu L, Yang D (2016) Modeling and simulation of the car-truck heterogeneous traffic flow based on a nonlinear car-following model. Appl Math Comput 273:706–717

    Google Scholar 

  • Muhammad SA, Qaisar I, Majid A, Shamrooz S (2023) Adaptive event-triggered robust H∞ control for Takagi-Sugeno fuzzy networked Markov jump systems with time-varying delay. Asian J Control 25(1):213–228

    Article  MathSciNet  Google Scholar 

  • Nguyen VL, Hwang RH, Lin PC, Vyas A, Nguyen VT (2022) Towards the age of intelligent vehicular networks for connected and autonomous vehicles in 6G. IEEE Netw. https://doi.org/10.1109/MNET.010.2100509

    Article  Google Scholar 

  • Payalan YF, Guvensan MA (2019) Towards next-generation vehicles featuring the vehicle intelligence. IEEE Trans Intell Transp Syst 21(1):30–47

    Article  Google Scholar 

  • Qaisar I, Majid A, Ramaraj P (2021) Design of sliding mode controller for sensor/actuator fault with unknown input observer for satellite control system. Soft Comput 25(24):14993–15003

    Article  Google Scholar 

  • Qian CF (2022) On the Design and validation of an autonomous vehicle perception system for the SAE/GM autodrive challenge II (DOCTORAL dissertation, University of Toronto (Canada)).

  • Sankaranarayanan R, Umadevi KS, Bhavani NPG, Jos BM, Haldorai A, Babu DV (2022) Cluster-based attacks prevention algorithm for autonomous vehicles using machine learning algorithms. Comput Electr Eng 101:108088

    Article  Google Scholar 

  • Singh D, Srivastava R (2022) Graph neural network with RNNs based trajectory prediction of dynamic agents for autonomous vehicle. Appl Intell 52(11):12801–12816

    Article  Google Scholar 

  • Tan L, Yu K, Lin L, Cheng X, Srivastava G, Lin JCW, Wei W (2021) Speech emotion recognition enhanced traffic efficiency solution for autonomous vehicles in a 5G-enabled space–air–ground integrated intelligent transportation system. IEEE Trans Intell Transp Syst 23(3):2830–2842

    Article  Google Scholar 

  • Tuncali CE, Fainekos G, Ito H, Kapinski J (2018) Simulation-based adversarial test generation for autonomous vehicles with machine learning components. In 2018 IEEE Intelligent Vehicles Symposium (IV) (pp. 1555–1562). IEEE

  • Ullah R, Dai X, Sheng A (2020a) Event-triggered scheme for fault detection and isolation of non-linear system with time-varying delay. IET Control Theory Appl 14(16):2429–2438

    Article  MathSciNet  Google Scholar 

  • Ullah R, Li Y, Aslam MS, Sheng A (2020b) Event-triggered dissipative observer-based control for delay dependent T-S fuzzy singular systems. IEEE Access 8:134276–134289

    Article  Google Scholar 

  • Wang S, Li C, Ng DWK, Eldar YC, Poor HV, Hao Q, Xu C (2022a) Federated deep learning meets autonomous vehicle perception: Design and verification. IEEE Netw 37:16–25

    Article  Google Scholar 

  • Wang FY, Song R, Zhou R, Wang X, Chen L, Li L, Zeng L, Zhou J, Teng S, Zhu X (2022b) Verification and validation of intelligent vehicles: objectives and efforts from China. IEEE Trans Intell Veh 7(2):164–169

    Article  Google Scholar 

  • Wu Q, Li X, Wang K et al (2023) Regional feature fusion for on-road detection of objects using camera and 3D-LiDAR in high-speed autonomous vehicles. Soft Comput 27:18195–18213. https://doi.org/10.1007/s00500-023-09278-3

    Article  Google Scholar 

  • Zhenhua M, Ullah R, Li Y, Sheng A, Majid A (2022) Stability and admissibility analysis of T-S descriptive systems and its applications. Soft Comput 26(15):7159–7166

    Article  Google Scholar 

Download references

Funding

No funding was provided for the completion of this study.

Author information

Authors and Affiliations

  1. Law and Public Management School, Leshan Normal University, Leshan, 614000, Sichuan, China

    Yan Zhang

  2. Faculty of Education Sciences, Leshan Normal University, Leshan, 614000, Sichuan, China

    Qianjun Tang

Authors
  1. Yan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qianjun Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qianjun Tang.

Ethics declarations

Conflict of interest

No conflict of interest has been declared by the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y., Tang, Q. Accelerating autonomy: an integrated perception digital platform for next generation self-driving cars using faster R-CNN and DeepLabV3. Soft Comput 28, 1633–1652 (2024). https://doi.org/10.1007/s00500-023-09510-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-023-09510-0

Keywords

Search

Navigation