Abstract
This paper presents a software-based vehicle detection and classification system capable of classifying traffic into four different classes, namely two-wheeler, three-wheeler, car, and heavy motor vehicle. It uses traffic video collected by a camera mounted on a vehicle parked by the side of a two-lane undivided road. Video frames containing vehicles are identified, both by automatically detecting peaks in Short Time Energy (STE) of the corresponding audio signal and adaptive background subtraction of the video frames, followed by blob subtraction and morphological operations. This may result in multiple images containing the same vehicle, which is eliminated using a Speeded Up Robust Feature (SURF) matching algorithm. Classification of resulting images is attempted in three different ways. In System 1, an SVM trained with explicit features such as Histogram of Gradient (HOG), Local Binary Pattern (LBP), and KAZE are used as a classifier and their performance is compared. In System 2, the task is performed using a deep neural network namely Single Shot Multibox Detector (SSD). The accuracy of the SSD system deteriorates when it is tested using video collected by another camera in a different environment. This issue is addressed in System 3 by retraining the SSD in the new set of images, without the use of manually labeled images. The effectiveness of all the proposed systems is validated using the collected heterogeneous traffic data.
Similar content being viewed by others
Data Availability
The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.
References
Adu-Gyamfi YO, Asare SK, Sharma A, Titus T (2017) Automated vehicle recognition with deep convolutional neural networks. Transp Res Rec 2645 (1):113–122
Alcantarilla PF, Bartoli A, Davison AJ (2012) Kaze features. In: European conference on computer vision. Springer, pp 214–227
Almusaylim ZA, Zaman N, Jung LT (2018) Proposing a data privacy aware protocol for roadside accident video reporting service using 5G in vehicular cloud networks environment. In: 2018 4th International conference on computer and information sciences (ICCOINS), Kuala Lumpur. IEEE, pp 1–5
Arróspide J, Salgado L (2014) A study of feature combination for vehicle detection based on image processing. Sci World J 2014
Ban XJ, Sun Z et al (2013) Vehicle classification using mobile sensors. University Transportation Research Center, pp 164–174
Barth A, Franke U (2009) Estimating the driving state of oncoming vehicles from a moving platform using stereo vision. IEEE Trans Intell Transp Syst 10 (4):560–571
Bay H, Tuytelaars T, Van Gool L (2006) SURF: speeded up robust features. In: European conference on computer vision. Springer, pp 404–417
Beke M, Haritaoglu E, Davis LS (2000) Real-time multiple vehicle detection and tracking from a moving vehicle. Mach Vis Appl 12(2):69–83
Chellappa R, Qian G, Zheng Q (2004) Vehicle detection and tracking using acoustic and video sensors. In: IEEE international conference on acoustics, speech, and signal processing, Montreal, Canada, vol 3, pp 790–793
Chen Y, Wu Q (2015) Moving vehicle detection based on optical flow estimation of edge. In: International conference on natural computation, Zhangjiajie, China, pp 754–758
Chen Z, Ellis T, Velastin SA (2011) Vehicle type categorization: a comparison of classification schemes. In: International IEEE conference on intelligent transportation systems, pp 74–79
Chen Z, Ellis T, Velastin SA (2012) Vehicle detection, tracking and classification in urban traffic. In: International IEEE conference on intelligent transportation systems, pp 951–956
Cheon M, Lee W, Yoon C, Park M (2012) Vision-based vehicle detection system with consideration of the detecting location. IEEE Trans Intell Transp Syst 13(3):1243–1252
Cheung SY, Coleri S, Dundar B, Ganesh S, Tan C-W, Varaiya P (2005) Traffic measurement and vehicle classification with single magnetic sensor. Transport Res Record Univ California, Berkeley 1917(1):173–181
Chung J, Sohn K (2017) Image-based learning to measure traffic density using a deep convolutional neural network. IEEE Trans Intell Transp Syst 19 (5):1670–1675
Cireşan DC, Meier U, Gambardella LM, Schmidhuber J (2010) Deep, big, simple neural nets for handwritten digit recognition. Neural Comput 22(12):3207–3220
Czapla Z (2016) Video-based vehicle detection on a two-way road. Zeszyty Naukowe Transport:11–22
Daniel C, Mary L (2016) Fusion of audio visual cues for vehicle classification. In: International conference on next generation intelligent systems, Kottayam, India, pp 1–4
Das J, Shah M, Mary L (2017) Bag of feature approach for vehicle classification in heterogeneous traffic. In: IEEE international conference on signal processing, informatics, communication and energy systems, Kollam, India, pp 1–5
Dedeoglu Y, Toreyin BU, Gudukbay U, Cetin AE (2008) Surveillance using both video and audio. In: Multimodal processing and interaction. Springer, New York, pp 1–13
Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The PASCAL visual object classes (VOC) challenge. Int J Comput Vis 88 (2):303–338
Fang J, Meng H, Zhang H, Wang X (2007) A low-cost vehicle detection and classification system based on unmodulated continuous-wave radar. In: IEEE intelligent transportation systems conference, Seattle, pp 715–720
George J, Cyril A, Koshy BI, Mary L (2013) Exploring sound signature for vehicle detection and classification using ANN. Int J Soft Comput 4 (2):29–36
George J, Mary L, Riyas K (2013) Vehicle detection and classification from acoustic signal using ANN and KNN. In: International conference on control communication and computing, Thiruvananthapuram, India, pp 436–439
Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 580–587
He K, Zhang X, Ren S, Sun J (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 37(9):1904–1916
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
Hoiem D, Rother C, Winn J (2007) 3D layout CRF for multi-view object class recognition and segmentation. In: IEEE conference on computer vision and pattern recognition, Minneapolis, pp 1–8
Hu Q, Wang H, Li T, Shen C (2017) Deep CNNs with spatially weighted pooling for fine-grained car recognition. IEEE Trans Intell Transp Syst 18 (11):3147–3156
Humayun M, Almufareh MF, Jhanjhi NZ (2022) Autonomous traffic system for emergency vehicles. Electronics 11(4):510
Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2017) Squeezenet: alexnet-level accuracy with 50x fewer parameters and < 0.5 mb model size. arXiv:1602.07360, 1–13
Kazemi FM, Samadi S, Poorreza HR, Akbarzadeh-t M-R (2007) Vehicle recognition based on fourier, wavelet and curvelet transforms-a comparative study. In: Fourth international conference on information technology, Las Vegas, pp 939–940
Kim H, Song B (2013) Vehicle recognition based on radar and vision sensor fusion for automatic emergency braking. In: International conference on control, automation and systems, Gwangju, Korea, pp 1342–1346
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105
LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324
LeCun Y, Huang FJ, Bottou L (2004) Learning methods for generic object recognition with invariance to pose and lighting. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, vol 2. Washington, DC, pp 100–104
Leutenegger S, Chli M, Siegwart R (2011) BRISK: binary robust invariant scalable keypoints. In: IEEE international conference on computer vision, pp 2548–2555
Li S, Yu H, Zhang J, Yang K, Bin R (2013) Video-based traffic data collection system for multiple vehicle types. IET Intell Transp Syst 8(2):164–174
Liu W, Wen X, Duan B, Yuan H, Wang N (2007) Rear vehicle detection and tracking for lane change assist. In: IEEE intelligent vehicles symposium, pp 252–257
Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg AC (2016) SSD: single shot multibox detector. In: European conference on computer vision, Germany. Springer, pp 21–37
Lowe G (2004) SIFT-the scale invariant feature transform. Int J 2:91–110
Ma X, Grimson WEL (2005) Edge-based rich representation for vehicle classification. In: Tenth IEEE international conference on computer vision, Washington, vol 2, pp 1185–1192
Ma Z, Chang D, Xie J, Ding Y, Wen S, Li X, Si Z, Guo J (2019) Fine-grained vehicle classification with channel max pooling modified CNNs. IEEE Trans Veh Technol 68(4):3224–3233
Mian ZF, Foss RW (2013) Vehicle evaluation using infrared data. US Patent 8,478,480, Google Patents. US Patent 8,478,480
Mikolajczyk K, Schmid C (2005) A performance evaluation of local descriptors. IEEE Trans Pattern Anal Mach Intell 27(10):1615–1630
Mishra PK, Banerjee B (2013) Vehicle classification using density based multi-feature approach in support vector machine classifier. Int J Comput Appl, vol 71(7)
Mithun NC, Rashid NU, Rahman SM (2012) Detection and classification of vehicles from video using multiple time-spatial images. IEEE Trans Intell Transp Syst 13(3):1215–1225
Nooralahiyan A, Kirby HR, McKeown D (1998) Vehicle classification by acoustic signature. Math Comput Model 27(9–11):205–214
Ntalampiras S (2018) Moving vehicle classification using wireless acoustic sensor networks. IEEE Trans Emerg Top Comput Intell 2(2):129–138
Paszke A, Chaurasia A, Kim S, Culurciello E (2016) Enet: a deep neural network architecture for real-time semantic segmentation. arXiv:1606.02147, 1–13
Piyush P, Rajan R, Mary L, Koshy BI (2016) Vehicle detection and classification using audio-visual cues. In: International conference on signal processing and integrated networks, pp 726–730
Prasad SA, Mary L (2019) A comparative study of different features for vehicle classification. In: 2019 International conference on computational intelligence in data science (ICCIDS). IEEE, Chennai, India, pp 1–5
Qiong W, Liao S-B (2017) Single shot multibox detector for vehicles and pedestrians detection and classification. DEStech Transactions on Engineering and Technology Research, Lancaster, pp 22–23
Rachmadi RF, Uchimura K, Koutaki G, Ogata K (2018) Single image vehicle classification using pseudo long short-term memory classifier. J Vis Commun Image Represent 56:265–274
Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788
Ren S, He K, Girshick R, Sun J (2015) Faster r-CNN: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99
Renganathan D, Mary L, George A (2018) Detection and classification of vehicles from heterogeneous traffic video data collected using a probe vehicle. In: 2018 3rd IEEE international conference on recent trends in electronics, information & communication technology (RTEICT). IEEE, Bengaluru, India, pp 1979–1982
Russakovsky O, Deng J, Krause J, Berg A, Fei-Fei L (2013) Large scale visual recognition challenge 2013. In: Proceedings of the international conference of computer vision (ICCV), pp 1–8
Selbes B, Sert M (2017) Multimodal vehicle type classification using convolutional neural network and statistical representations of MFCC. In: IEEE international conference on advanced video and signal based surveillance, pp 1–6
Sermanet P, Eigen D, Zhang X, Mathieu M, Fergus R, LeCun Y (2013) Overfeat: integrated recognition, localization and detection using convolutional networks. arXiv:1312.6229, pp 2–20
Shen Y, Xiao T, Li H, Yi S, Wang X (2017) Learning deep neural networks for vehicle re-id with visual-spatio-temporal path proposals. In: Proceedings of the IEEE international conference on computer vision, pp 1900–1909
Shvai N, Hasnat A, Meicler A, Nakib A (2019) Accurate classification for automatic vehicle-type recognition based on ensemble classifiers. IEEE Trans Intell Transp Syst 21(3):1288–1297
Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 1–14
Sivaraman S, Trivedi MM (2012) Real-time vehicle detection using parts at intersections. In: International IEEE conference on intelligent transportation systems, pp 1519–1524
Sivaraman S, Trivedi MM (2013) Looking at vehicles on the road: a survey of vision-based vehicle detection, tracking, and behavior analysis. IEEE Trans Intell Transp Syst 14(4):1773–1795
Song K, Chen C, Huang C (2004) Design and experimental study of an ultrasonic sensor system for lateral collision avoidance at low speeds. In: IEEE intelligent vehicles symposium, Parma, pp 647–652
Suhao L, Jinzhao L, Guoquan L, Tong B, Huiqian W, Yu P (2018) Vehicle type detection based on deep learning in traffic scene. Procedia Comput Sci 131:564–572
Szegedy C, Toshev A, Erhan D (2013) Deep neural networks for object detection. In: Advances in neural information processing systems, pp 2553–2561
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9
Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-first AAAI conference on artificial intelligence, pp 4278–4284
Wang T, Zhu Z (2012) Multimodal and multi-task audio-visual vehicle detection and classification. In: IEEE ninth international conference on advanced video and signal-based surveillance, pp 440–446
Wang C, Thorpe C, Suppe A (2003) Ladar-based detection and tracking of moving objects from a ground vehicle at high speeds. In: IEEE IV2003 intelligent vehicles symposium. Proceedings, Columbus, pp 416–421
Wang K, Wang R, Feng Y, Zhang H, Huang Q, Jin Y, Zhang Y (2014) Vehicle recognition in acoustic sensor networks via sparse representation. In: IEEE international conference on multimedia and expo workshops (ICMEW), Chengdu, pp 1–4
Wen X, Shao L, Xue Y, Fang W (2015) A rapid learning algorithm for vehicle classification. Inf Sci 295:395–406
Zhang R, Ding J (2012) Object tracking and detecting based on adaptive background subtraction. Procedia Eng 29:1351–1355
Acknowledgements
The authors would like to thank the Transportation Research Center, College of Engineering, Thiruvananthapuram (Government of Kerala), India, for the support provided for this work.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
S., A.P., Mary, L. & Koshy, B.I. Detection and classification of vehicles using audio visual cues. Multimed Tools Appl 82, 44087–44106 (2023). https://doi.org/10.1007/s11042-023-14868-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-023-14868-2