Skip to main content
SpringerLink
Log in

Accelerated and optimized covariance descriptor for pedestrian detection in self-driving cars

  • Published:
Design Automation for Embedded Systems Aims and scope Submit manuscript

Abstract

Self-Driving vehicles are expected to thrive in the coming years. These vehicles are designed to analyze the environment around them in real-time to identify obstacles and hazards. One of the most important aspects of designing a self-driving vehicle is to preserve the safety of pedestrians. This requires accurate and rapid pedestrian detection, which is a key operation in various other applications including video surveillance and assisted living. The covariance descriptor is one of the most efficient descriptors used in detecting pedestrians. However, the descriptor is compute-intensive; rendering it less favorable for real-time applications. This paper proposes an accelerated and optimized implementation of the descriptor. Instead of mapping the entire descriptor to a hardware accelerator, we opt for a heterogeneous architecture. In particular, compute-intensive components of the descriptor are accelerated on hardware, while the other components are executed on an embedded processor. The proposed architecture combines both speed and flexibility while being watchful of precious hardware resources. This architecture was validated on a Zynq SoC platform, which hosts FPGA fabric along with an ARM processor. The results of executing the descriptor on the platforms show a performance gain of up to 13.52 × when compared to pure software implementation of the descriptor.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Mikolajczyk K, Schmid C (2003) A performance evaluation of local descriptors. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2003, pp 525–531

  2. Bay H, Tuytelaars T, Van Gool L (2006) Surf: Speeded up robust features. In: European conference on computer vision. Springer, Berlin, pp 404–417

  3. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection (2005). In: IEEE computer society conference on computer vision and pattern recognition, pp 886–893

  4. Rani R, Kumar R, Singh AP (2016) A comparative study of object recognition techniques. In: Proceedings of 7th international conference on intelligent systems, modelling and simulation (ISMS)

  5. Tuzel O, Porikli F, Meer P (2006) Region covariance: a fast descriptor for detection and classification. In: Proceedings of European conference on computer vision ECCV, Graz, Austria, May 2006, pp 589–600

  6. Oncel T, Fatih P, Peter M (2008) Pedestrian detection via classification on Riemannian manifolds. IEEE Trans Pattern Anal Mach Intell 30(10):1713–1727

    Article  Google Scholar 

  7. y Terán ARM, Lacassagne L, Zahraee AH, Gouiffàs M (2013) Real-time covariance tracking algorithm for embedded systems. In: 2013 Conference on design and architectures for signal and image processing. IEEE, pp 104–111

  8. Romero A, Lacassagne L, Gouiffès M, Zahraee AH (2014) Covariance tracking: architecture optimizations for embedded systems. EURASIP J Adv Signal Process 2014(1):175

    Article  Google Scholar 

  9. Abid N, Ouni T, Loukil K, Abid M, Ammeri AC (2016) Multiprocessor architecture for an optimazed parallel model of covariance based pedestrian detection. In: 2016 11th International design & test symposium (IDT). IEEE, pp 71–76

  10. Abid N, Loukil K, Ouni T, Ayedi W, Ammari AC, Abid M (2020) An improvement of multi-scale covariance descriptor for embedded system. J Real-Time Image Proc 17(3):419–435

    Article  Google Scholar 

  11. Martelli S, Tosato D, Cristani M, Murino V (2011) FPGA-based pedestrian detection using array of covariance features. In: 2011 5th ACM/IEEE international conference on distributed smart cameras, ICDSC 2011

  12. Martelli S, Tosato D, Cristani M, et al (2011) Fast FPGA-based architecture for pedestrian detection based on covariance matrices. In: Proceedings of IEEE international conference on image processing, Brussels, September 2011, pp 389–392

  13. Abid N, Ayedi W, Abid M, Ammari AC (2014) SW/HW implementation of image covariance descriptor for pedestrian detection systems. In: 2014 1st International conference on advanced technologies for signal and image processing (ATSIP). IEEE, pp 115–119

  14. Said Y, Atri M (2016) Efficient and high-performance pedestrian detector implementation for intelligent vehicles. IET Intel Transp Syst 10(6):438–444

    Article  Google Scholar 

  15. Safaei A, Wu QJ, Akilan T (2017) System-on-chip-based hardware acceleration for human detection in 2d/3d scenes. In: 2017 IEEE international conference on systems, man, and cybernetics (SMC). IEEE, pp 1041–1045

  16. Port O, Etsion Y (2017) DFiant: a dataflow hardware description language. In: 2017 27th International conference on field programmable logic and applications (FPL). IEEE, pp 1–4

  17. https://www.xilinx.com/support/documentation/sw_manuals/xilinx2019_2/ug902-vivado-high-levelsynthesis.pdf. Accessed on 29 March 2021

  18. Khriji L, Ammari AC, Awadalla M (2020) HW/SW co-design for dates classification on Xilinx Zynq SoC. In: Proceedings of the 26th conference of open innovations association FRUCT, Yaroslavl, Russia, April 20–24, 2020

  19. Krizhevsky A, Sutskever I, Hinton GE (2012) Image Net Classification with Deep Convolutional Neural Networks. In: Proceedings of advances in neural information processing systems, vol 5, pp 1106–1114

  20. Girshick R (2015) Fast R-CNN. In: Proceedings of the 2015 IEEE international conference on computer vision (ICCV). IEEE, pp 1440–1448

  21. Liu W et al (2016) SSD: single shot MultiBox detector. In: Leibe B, Matas J, Sebe N, Welling M (eds) Computer vision—ECCV 2016. ECCV 2016. Lecture notes in computer science, vol 9905. Springer

  22. Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 779–788).

  23. Song X, Gao S, Chen C (2021) A multispectral feature fusion network for robust pedestrian detection. Alex Eng J 60(1):73–85

    Article  Google Scholar 

  24. Ng CJ, Low CY, Toh KA, Kim J, Teoh ABJ (2018) Orthogonal filter banks with region Log-TiedRank covariance matrices for face recognition. J Vis Commun Image Represent 55:548–560

    Article  Google Scholar 

  25. Tsai CY, Chung WH (2018) Multiclass object classification using covariance descriptors with kernel SVM. J Comput 29(5):244–249

    Google Scholar 

  26. Ren J, Wu XJ, Kittler J (2020) Discriminative block-diagonal covariance descriptors for image set classification. Pattern Recogn Lett 136:230–236

    Article  Google Scholar 

  27. Hu Y, Wang Z, AlRegib G (2020) Texture classification using block intensity and gradient difference (BIGD) descriptor. Signal Process: Image Commun 1:115770

    Google Scholar 

  28. Guo C, Zhang Z, Li J, Jiang X, Zhang J, Zhang L (2020) Robust visual tracking using kernel sparse coding on multiple covariance descriptors. ACM Trans Multimed Comput Commun Appl (TOMM) 16(1):1–22

    Google Scholar 

  29. Abid N, Ouni T, Ammari AC, Abid M (2022) Efficient and high-performance pedestrian detection implementation for intelligent vehicles. Multimed Syst 28(1):69–84

    Article  Google Scholar 

  30. Fang L, Wu G, Kang W, Wu Q, Wang Z, Feng DD (2019) Feature covariance matrix-based dynamic hand gesture recognition. Neural Comput Appl 31(12):8533–8546

    Article  Google Scholar 

  31. Boutellaa E, Kerdjidj O, Ghanem K (2019) Covariance matrix based fall detection from multiple wearable sensors. J Biomed Inform 94:103189

    Article  Google Scholar 

  32. Talebi H, Ranjbar A, Davoudi A, Gholami H, Menhaj MB (2019) High accuracy classification of white blood cells using TSLDA classifier and covariance features. arXiv preprint arXiv:1906.05131

  33. Pu L, Feng X, Hou Z (2019) Covariance intersection fusion for visual tracking with hierarchical features. In: 10th International conference on graphics and image processing (ICGIP 2018), vol 11069. International Society for Optics and Photonics, pp 1106941

  34. Zhang W, Yan Z, Wang Q, Wu X, Zuo W (2020) Learning second-order statistics for place recognition based on robust covariance estimation of CNN features. Neurocomputing

  35. Otberdout N, Kacem A, Daoudi M, Ballihi L, Berretti S (2019) Automatic analysis of facial expressions based on deep covariance trajectories. IEEE Trans Neural Networks Learn Syst 31(10):3892–3905

    Article  Google Scholar 

  36. Suresh P, Saravanakumar U, Celestine I, Senthilkumar M, Gautam S (2021) Field-programmable gate arrays in a low power vision system. Comput Electr Eng 90(2021):106996

    Google Scholar 

  37. Bagni D, Kannan P, Neuendorffer S (2017) Demystifying the Lucas–Kanade optical flow algorithm with Vivado HLS, XAPP1300 (v1.0) 3 February 2017

  38. ZC702 Evaluation Board for the Zynq-7000 XC7Z020 SoC, UG850

  39. Mewada HK, Patel AV, Mahant KK (2018) Concurrent design of active contour for image segmentation using Zynq ZC702. Comput Electr Eng 72:631–643

    Article  Google Scholar 

  40. Khriji L, Ammari AC, Awadalla M (2020) Hardware/software co-design of a vision system for automatic classification of date fruits. Int J Embed Real-Time Commun Syst (IJERTCS) 11(4):21–40

    Article  Google Scholar 

  41. Messaoud S, Bouaafia S, Maraoui A, Ammari AC, Khriji L, Machhout M (2022) Deep convolutional neural networks-based Hardware-Software on-chip system for computer vision application. Comput Electr Eng 98:107671

    Article  Google Scholar 

  42. Xilinx Documentation SDSoC Environment User Guide UG1027. Available online: https://www.xilinx.com/support/documentation/sw_manuals/xilinx2017_4/ug1027-sdsoc-user-guide.pdf. Accessed on 29 March 2021

  43. Ammari AC, Jemai A (2009) Multiprocessor Platform Based Design for Multimedia. IET Comput Digital Tech 3(1):52–61

    Article  Google Scholar 

  44. Benammar M, Alassi A, Gastli A, Ben-Brahim L, Touati F (2019) New fast arctangent approximation algorithm for generic real-time embedded applications. Sensors 19(23):5148

    Article  Google Scholar 

  45. Pilato L, Fanucci L, Saponara S (2017) Real-time and high-accuracy arctangent computation using CORDIC and fast magnitude estimation. Electronics 6:22–30

    Article  Google Scholar 

  46. Schwarzbacher AT, Brasching A, Wahl TH, Foley PCB (1999) Optimizations and implementation of the arctan function for the power domain. In: Proceedings of the second electronic circuits and systems conference, p 33

  47. Abramowitz M, Stegun IA (eds) (1972). Dover, New York

    MATH  Google Scholar 

  48. Rajan S, Wang S, Inkol R (2006) Efficient approximations for the arctangent function. IEEE Signal Process Mag 23:108–111

    Article  Google Scholar 

  49. Lomont, C. (2003). Fast inverse square root. Technical Report. http://www.lomont.org/Math/Papers/2003/InvSqrt.pdf

  50. Zhang N, Wei X, Chen H, Liu W (2021) FPGA implementation for CNN-based optical remote sensing object detection. Electronics 10(3):282

    Article  Google Scholar 

  51. Cai L, Wang C, Xu Y (2021) A real-time FPGA accelerator based on winograd algorithm for underwater object detection. Electronics 10(23):2889

    Article  Google Scholar 

  52. Baig MA, Pasha MA, Masud S (2020) An optimized hardware/software co-design framework for real-time pedestrian detection. In: 2020 International conference on electrical, communication, and computer engineering (ICECCE). IEEE, pp 1–6

  53. Dürre J, Paradzik D, Blume H (2018) A HOG-based real-time and multi-scale pedestrian detector demonstration system on FPGA. In: Proceedings of the 2018 ACM/SIGDA international symposium on field-programmable gate arrays, pp 163–172

Download references

Acknowledgements

This project was funded by Sultan Qaboos University (SQU), Deanship of Scientific Research (DSR), under Grant No. “IG/ENG/ECED/19/01”. The authors, therefore, acknowledge and thanks SQU for its financial support.

Author information

Authors and Affiliations

  1. Laboratory of Computer and Embedded Systems, National School of Engineering of Sfax, Sfax University, Sfax, Tunisia

    Nesrine Abid & Mohamed Abid

  2. Department of Electrical and Computer Engineering, College of Engineering, Sultan Qaboos University, Al-khod123, Muscat, Sultanate of Oman

    Ahmed. C. Ammari & Ahmed Al Maashri

  3. Department of Computers and systems, Helwan University, Helwan, Cairo, Egypt

    Medhat Awadalla

Authors
  1. Nesrine Abid
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ahmed. C. Ammari
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ahmed Al Maashri
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mohamed Abid
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Medhat Awadalla
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ahmed. C. Ammari.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Abid, N., Ammari, A.C., Al Maashri, A. et al. Accelerated and optimized covariance descriptor for pedestrian detection in self-driving cars. Des Autom Embed Syst 27, 139–163 (2023). https://doi.org/10.1007/s10617-023-09273-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10617-023-09273-9

Keywords

Search

Navigation