Deep multiple classifier fusion for traffic scene recognition

Abstract

The recognition of the traffic scene in still images is an important yet difficult task in an intelligent transportation systems. The main difficulty lies in how to improve the image processing algorithms for different traffic participants and the various layouts of roads while discriminating the different traffic scenes. In this paper, we attempt to solve the traffic scene recognition problem with three distinct contributions. First, we propose a deep multi-classifier fusion method in the setting of granular computing. Specifically, the deep multi-classifier fusion method which involves local deep-learned feature extraction at one end that is connected to the other end for classification through a multi-classifier fusion approach. At the local deep-learned feature extraction end, the operation of convolution to extract feature maps from the local patches of an image is essentially a form of information granulation, whereas the fusion of classifiers at the classification end is essentially a form of organization. The second contribution is the creation of new traffic scene data set, named the “WZ-traffic”. The WZ-traffic data set consists of 6035 labeled images, which belong to 20 categories collected from both an image search engine as well as from personal photographs. Third, we make extensive comparisons with state-of-the-art methods on the WZ-traffic and FM2 data sets. The experiment results demonstrate that our method dramatically improves traffic scene recognition and brings potential benefits to many other real-world applications.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

References

  1. Abdi H, Williams LJ (2010) Principal component analysis. Wiley Interdiscipl Reviews Comput Stat 2(4):433–459

    Article  Google Scholar 

  2. Altman NS (1992) An introduction to kernel and nearest-neighbor nonparametric regression. Am Stat 46(3):175–185

    MathSciNet  Google Scholar 

  3. Atanassov KT (1999) Interval valued intuitionistic fuzzy sets. In: Kacprzyk J (ed) Studies in fuzziness and soft computing, vol 35. Physica, Heidelberg

    Google Scholar 

  4. Chen CY, Choi W, Chandraker M (2016) Atomic scenes for scalable traffic scene recognition in monocular videos. In: 2016 IEEE winter conference on applications of computer vision (WACV), IEEE, pp 1–9

  5. Chen SM, Chang YC (2011) Weighted fuzzy rule interpolation based on ga-based weight-learning techniques. IEEE Trans Fuzzy Syst 19(4):729–744

    Article  Google Scholar 

  6. Chen SM, Han WH (2018) An improved madm method using interval-valued intuitionistic fuzzy values. Inf Sci 467:489–505

    MathSciNet  Article  Google Scholar 

  7. Chen SM, Huang CM (2003) Generating weighted fuzzy rules from relational database systems for estimating null values using genetic algorithms. IEEE Trans Fuzzy Syst 11(4):495–506

    Article  Google Scholar 

  8. Chen SM, Tanuwijaya K (2011) Fuzzy forecasting based on high-order fuzzy logical relationships and automatic clustering techniques. Expert Syst Appl 38(12):15425–15437

    Article  Google Scholar 

  9. Chen SM, Wang JY (1995) Document retrieval using knowledge-based fuzzy information retrieval techniques. IEEE Trans Syst Man Cybern 25(5):793–803

    Article  Google Scholar 

  10. Chen SM, Lee LW, Liu HC, Yang SW (2012a) Multiattribute decision making based on interval-valued intuitionistic fuzzy values. Expert Syst Appl 39(12):10343–10351

    Article  Google Scholar 

  11. Chen SM, Yang MW, Yang SW, Sheu TW, Liau CJ (2012b) Multicriteria fuzzy decision making based on interval-valued intuitionistic fuzzy sets. Expert Syst Appl 39(15):12085–12091

    Article  Google Scholar 

  12. Cheng SH, Chen SM, Jian WS (2016) Fuzzy time series forecasting based on fuzzy logical relationships and similarity measures. Inf Sci 327:272–287

    MathSciNet  MATH  Article  Google Scholar 

  13. Cortes C, Vapnik V (1995) Support-Vector Networks. Kluwer Academic Publishers, Dordrecht

    Google Scholar 

  14. Csurka G, Perronnin F (2010) Fisher vectors: beyond bag-of-visual-words image representations. In: International conference on computer vision, imaging and computer graphics, Springer, pp 28–42

  15. Csurka G, Dance C, Fan L, Willamowski J, Bray C (2004) Visual categorization with bags of keypoints. In: Workshop on statistical learning in computer vision, ECCV, Prague 1, pp 1–2

  16. Cutler A, Cutler DR, Stevens JR (2004) Random forests. Mach Learn 45(1):157–176

    Google Scholar 

  17. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE Computer society conference on computer vision and pattern recognition, 2005. CVPR 2005, IEEE, vol 1, pp 886–893

  18. Dixit M, Chen S, Gao D, Rasiwasia N, Vasconcelos N (2015) Scene classification with semantic fisher vectors. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2974–2983

  19. Duin RP, Tax DM (1998) Classifier conditional posterior probabilities. In: Joint IAPR international workshops on statistical techniques in pattern recognition (SPR) and structural and syntactic pattern recognition (SSPR), Springer, pp 611–619

  20. Ejegwa PA (2018) Distance and similarity measures for pythagorean fuzzy sets. Granul Comput. https://doi.org/10.1007/s41066-018-00149-z

    Article  Google Scholar 

  21. Ess A, Müller T, Grabner H, Van Gool LJ (2009) Segmentation-based urban traffic scene understanding. BMVC 1:2

    Google Scholar 

  22. Friedman JH (2002) Stochastic gradient boosting. Comput Stat Data Anal 38(4):367–378

    MathSciNet  MATH  Article  Google Scholar 

  23. Gardner MW, Dorling S (1998) Artificial neural networks (the multilayer perceptron)? a review of applications in the atmospheric sciences. Atmos Environ 32(14–15):2627–2636

    Article  Google Scholar 

  24. Garg H, Kaur G (2018) Novel distance measures for cubic intuitionistic fuzzy sets and their applications to pattern recognitions and medical diagnosis. Granular Computing pp 1–16

  25. Girshick R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 1440–1448

  26. Gondy LA, Thomas CRB, Bayes N (1993) Programs for machine learning. Adv Neural Inf Process Syst 79(2):937–944

    Google Scholar 

  27. Greene MR, Botros AP, Beck DM, Fei-Fei L (2015) What you see is what you expect: rapid scene understanding benefits from prior experience. Attent Percept Psychophys 77(4):1239–1251

    Article  Google Scholar 

  28. Huang T, Koller D, Malik J, Ogasawara G, Rao B, Russell SJ, Weber J (1994) Automatic symbolic traffic scene analysis using belief networks. AAAI 94:966–972

    Google Scholar 

  29. Jégou H, Douze M, Schmid C, Pérez P (2010) Aggregating local descriptors into a compact image representation. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR), IEEE, pp 3304–3311

  30. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM international conference on multimedia, ACM, pp 675–678

  31. Khan MSA, Abdullah S, Ali A, Amin F, Rahman K (2018) Hybrid aggregation operators based on pythagorean hesitant fuzzy sets and their application to group decision making. Granul Comput. 4(3):469–482

    Article  Google Scholar 

  32. Krapac J, Verbeek J, Jurie F (2011) Modeling spatial layout with fisher vectors for image categorization. In: 2011 International Conference on Computer Vision, IEEE, pp 1487–1494

  33. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Neural Inf Process Syst. https://doi.org/10.1145/3065386

    Article  Google Scholar 

  34. Kuncheva LI (2002) A theoretical study on six classifier fusion strategies. IEEE Trans Pattern Anal Mach Intell 24(2):281–286

    Article  Google Scholar 

  35. Lee LW, Chen SM (2008) Fuzzy risk analysis based on fuzzy numbers with different shapes and different deviations. Expert Syst Appl 34(4):2763–2771

    Article  Google Scholar 

  36. Liu H, Cocea M (2018) Granular computing-based approach of rule learning for binary classification. Granul Comput 4(2):275–283

    Article  Google Scholar 

  37. Liu H, Gegov A (2015) Collaborative decision making by ensemble rule based classification systems. In: Pedrycz W, Chen S-M (eds) Granular computing and decision making: interactive and iterative approaches, vol 10. Springer, Cham, pp 245–264

    Google Scholar 

  38. Liu H, Gegov A, Cocea M (2016) Rule-based systems: a granular computing perspective. Granular Computing 1(4):259–274

    Article  Google Scholar 

  39. Liu H, Cocea M, Ding W (2018) Multi-task learning for intelligent data processing in granular computing context. Granul Comput 3(3):257–273

    Article  Google Scholar 

  40. Lu K, Li J, An X, He H (2016) Generalized haar filter based deep networks for real-time object detection in traffic scene. arXiv preprint arXiv:161009609

  41. Mandal P, Ranadive A (2018) Hesitant bipolar-valued fuzzy sets and bipolar-valued hesitant fuzzy sets and their applications in multi-attribute group decision making. Granul Comput 4(3):559–583

    Article  Google Scholar 

  42. Mioulet L, Breckon TP, Mouton A, Liang H, Morie T (2013) Gabor features for real-time road environment classification. In: Industrial Technology (ICIT), 2013 IEEE International Conference on, IEEE, pp 1117–1121

  43. Nanni L, Lumini A (2013) Heterogeneous bag-of-features for object/scene recognition. Appl Soft Comput 13(4):2171–2178

    Article  Google Scholar 

  44. Oliva A, Torralba A (2001) Modeling the shape of the scene: a holistic representation of the spatial envelope. Int J Comput Vis 42(3):145–175

    MATH  Article  Google Scholar 

  45. Oquab M, Bottou L, Laptev I, Sivic J (2014) Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1717–1724

  46. Pedrycz W (2011) Information granules and their use in schemes of knowledge management. Sci Iran 18(3):602–610

    Article  Google Scholar 

  47. Pedrycz W, Chen SM (2015) Granular computing and decision-making: interactive and iterative approaches, vol 10. Springer, New York

    Google Scholar 

  48. Sikiric I, Brkic K, Krapac J, Segvic S (2014) Image representations on a budget: traffic scene classification in a restricted bandwidth scenario. In: Intelligent vehicles symposium proceedings, IEEE, pp 845–852

  49. Simon M, Rodner E, Denzler J (2014) Part detector discovery in deep convolutional neural networks. In: Asian conference on computer vision, Springer, pp 162–177

  50. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14091556

  51. Song X, Jiang S, Herranz L (2015) Joint multi-feature spatial context for scene recognition on the semantic manifold. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1312–1320

  52. Tang I, Breckon TP (2011) Automatic road environment classification. IEEE Trans Intell Transport Syst 12(2):476–484

    Article  Google Scholar 

  53. Taylor P, Griffiths N, Bhalerao A, Anand S, Popham T, Xu Z, Gelencser A (2016) Data mining for vehicle telemetry. Appl Artif Intell 30(3):233–256

    Article  Google Scholar 

  54. Toufiq R, Isalm MR (2017) Face recognition system using soft-output classifier fusion method. In: International conference on electrical, computer and telecommunication engineering, pp 1–4

  55. Wang J, Yang J, Yu K, Lv F, Huang T, Gong Y (2010) Locality-constrained linear coding for image classification. In: 2010 IEEE computer society conference on computer vision and pattern recognition, Citeseer, pp 3360–3367

  56. Wu F (2019) WZ-traffic dataset. https://github.com/Fangyu0505/traffic-scene-recognition. Accessed 10 Mar 2019

  57. Wu F, Yan S, Smith JS, Zhang B (2018) Joint semi-supervised learning and re-ranking for vehicle re-identification. In: IEEE international conference on pattern recognition, vol 1. IEEE, pp 278–283

  58. Wu F, Yan S, Smith JS, Zhang B (2019) Vehicle re-identification in still images: application of semi-supervised learning and re-ranking. Signal Process Image Commun 76:261–271

    Article  Google Scholar 

  59. Yan S, Smith JS, Zhang B (2017) Action recognition from still images based on deep vlad spatial pyramids. Signal Process Image Commun 54:118–129

    Article  Google Scholar 

  60. Yao J (2005a) Information granulation and granular relationships. In: 2005 IEEE international conference on granular computing, IEEE, vol 1, pp 326–329

  61. Yao Y (2005b) Perspectives of granular computing. In: 2005 IEEE international conference on granular computing, IEEE, vol 1, pp 85–90

  62. Zadeh LA (1965) Fuzzy sets. Inf Control 8(3):338–353

    MATH  Article  Google Scholar 

  63. Zavaschi THH, Britto AS Jr, Oliveira LES, Koerich AL (2013) Fusion of feature sets and classifiers for facial expression recognition. Expert Syst Appl 40(2):646–655

    Article  Google Scholar 

  64. Zhou ZH (2012) Ensemble methods: foundations and algorithms. Chapman & Hall, Boca Raton

    Google Scholar 

  65. Zitnick CL, Dollár P (2014) Edge boxes: locating object proposals from edges. In: European conference on computer vision, Springer, pp 391–405

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Fangyu Wu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Wu, F., Yan, S., Smith, J.S. et al. Deep multiple classifier fusion for traffic scene recognition. Granul. Comput. 6, 217–228 (2021). https://doi.org/10.1007/s41066-019-00182-6

Download citation

Keywords

  • Traffic scene recognition
  • Convolutional neural networks
  • Multi-classifier fusion