Convolution neural network joint with mixture of extreme learning machines for feature extraction and classification of accident images

  • Ali Pashaei
  • Mehdi GhateeEmail author
  • Hedieh Sajedi
Original Research Paper


This paper considers the accident images and develops a deep learning method for feature extraction together with a mixture of experts for classification. For the first task, the outputs of the last max-pooling layer of a Convolution Neural Network (CNN) are used to extract the hidden features automatically. For the second task, a mixture of advanced variations of Extreme Learning Machine (ELM) including basic ELM, constraint ELM (CELM), On-Line Sequential ELM (OSELM) and Kernel ELM (KELM), is developed. This ensemble classifier combines the advantages of different ELMs using a gating network and its accuracy is very high while the processing time is close to real-time. To show the efficiency, the different combinations of the traditional feature extraction and feature selection methods and the various classifiers are examined on two kinds of benchmarks including accident images’ data set and some general data sets. It is shown that the proposed system detects the accidents with 99.31% precision, recall and F-measure. Besides, the precisions of accident-severity classification and involved-vehicle classification are 90.27% and 92.73%, respectively. This system is suitable for on-line processing on the accident images that will be captured by Unmanned Aerial Vehicles (UAV) or other surveillance systems.


Feature extraction Accident images’ classification Convolutional neural networks Mixture of ELM Ensemble learning 



  1. 1.
    Bisht, N., Siddhi, P., Kashyap, H.: Monitoring road accidents using sensors and providing medical facilities. Treatise Electr Magn 2, 68–73 (2012)Google Scholar
  2. 2.
    Hoose, N., Vicencio, M., Zhang, X.: Incident detection in urban roads using computer image processing. Traffic Eng Control 33(4), 236–244 (1992)Google Scholar
  3. 3.
    Zifeng, J.: Macro and micro freeway automatic incident detection (aid) methods based on image processing. In: Intelligent Transportation System, ITSC’97 (1997)Google Scholar
  4. 4.
    Coifman, B., McCord, M., Mishalani, R., Iswalt, M., Ji, Y.: Roadway traffic monitoring from an unmanned aerial vehicle. In: IEE Proceedings-Intelligent Transport Systems (2006)Google Scholar
  5. 5.
    Cao, X., Lan, J., Yan, P., Li, X.: Vehicle detection and tracking in airborne videos by multi-motion layer analysis. Mach. Vis. Appl. 23(5), 921–935 (2012)CrossRefGoogle Scholar
  6. 6.
    Kim, N., Chervonenkis, M.: Situation control of unmanned aerial vehicles for road traffic monitoring. Modern Appl. Sci. 9(5), 1 (2015)Google Scholar
  7. 7.
    Srinivasan, D., Jin, X., Cheu, R.: Evaluation of adaptive neural network models for freeway incident detection. IEEE Trans. Intell. Transp. Syst. 5(1), 1–11 (2004)CrossRefGoogle Scholar
  8. 8.
    Chiou, Y.-C., Fu, C.: Modeling crash frequency and severity using multinomial-generalized poisson model with error components. Accid. Anal. Prev. 50, 73–82 (2013)CrossRefGoogle Scholar
  9. 9.
    Anderson, J., Govada, M., Steffen, T., Thorne, C., Varvarigou, V., Kales, S., Burks, S.: Obesity is associated with the future risk of heavy truck crashes among newly recruited commercial drivers. Accid. Anal. Prev. 49, 378–384 (2012)CrossRefGoogle Scholar
  10. 10.
    Ahonen, T., Hadid, A., Pietikainen M.: Face description with local binary patterns: application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 28(12), 2037–2041 (2006)CrossRefzbMATHGoogle Scholar
  11. 11.
    Wang, X., Han, T., Yan, S.: An hog-Lbp human detector with partial occlusion handling. In: International Conference on Computer Vision (2009)Google Scholar
  12. 12.
    Chen, H., Tsai, S., Schroth, G., Chen, D., Grzeszczuk, R., Girod, B.: Robust text detection in natural images with edge-enhanced maximally stable extremal regions. In: 18th IEEE International Conference on Image Processing (ICIP) (2011)Google Scholar
  13. 13.
    Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: Speeded-up robust features (surf). Comput Vis. Image Underst. 110(3), 346–359 (2008)CrossRefGoogle Scholar
  14. 14.
    Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1097–1105 (2012)Google Scholar
  15. 15.
    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Conference on computer vision and pattern recognition, pp. 779–788 (2016)Google Scholar
  16. 16.
    Ahn, B.: Real-time video object recognition using convolutional neural network. In: Neural Networks (IJCNN). pp. 1–7 (2015)Google Scholar
  17. 17.
    LeCun, Y., Kavukcuoglu, K., Farabet, C.: Convolutional networks and applications in vision. In: International Symposium on Circuits and Systems (ISCAS) (2010)Google Scholar
  18. 18.
    Lee, K., Park, D.C.: Image classification using fast learning convolutional neural networks. Adv. Sci. Technol. Lett. 113, 50–55 (2015)CrossRefGoogle Scholar
  19. 19.
    Sadeky, S., Al-Hamadiy, A., Michaelisy, B., Sayed, U.: “Real-time automatic traffic accident recognition using Hfg. In: 20th International Conference on Pattern Recognition (ICPR) (2010)Google Scholar
  20. 20.
    Nejjari, F., Benhlima, L., Bah S.: Event traffic detection using heterogenous wireless sensors network. In: 13th International Conference of Computer Systems and Applications (AICCSA) (2016)Google Scholar
  21. 21.
    Kahaki, S., Nordin, M.: Highway traffic incident detection using high-resolution aerial remote sensing imagery. J. Comput. Sci. 7(6), 949 (2011)CrossRefGoogle Scholar
  22. 22.
    Jiansheng, F.: Vision-based real-time traffic accident detection. In: 11th World Congress on Intelligent Control and Automation, WCICA (2014)Google Scholar
  23. 23.
    Chen, L., Cao, Y., Ji, R.: Automatic incident detection algorithm based on support vector machine. In: Sixth International Conference on Natural Computation (ICNC) (2010)Google Scholar
  24. 24.
    Prabha, C., Sunitha, R., Anitha, R.: Automatic vehicle accident detection and messaging system using GSM and GPS modem. Int. J. Adv. Res Electr. Electron. Instrum. Eng. 3(7), 10723–10727 (2014)Google Scholar
  25. 25.
    Kagesawa, M., Nakamura, A., Ikeuchi, K., Saito H.: Vehicle type classification in infra-red image using parallel vision board. ITSWC (2000)Google Scholar
  26. 26.
    Zhou, Y., Nejati, H., Do, T.-T., Cheung, N.-M., Cheah, L.: Image-based vehicle analysis using deep neural network: a systematic study. In: International Conference on Digital Signal Processing (DSP) (2016)Google Scholar
  27. 27.
    Chen, Z., Ellis, T.: Semi-automatic annotation samples for vehicle type classification in urban environments. IET Intel. Transp. Syst. 3(9), 240–249 (2014)Google Scholar
  28. 28.
    Wang, X., Zhang, W., Wu, X., Xiao, L., Qian, Y., Fang, Z.: Real-time vehicle type classification with deep convolutional neural networks. J. Real-Time Image Proc. (2017). Google Scholar
  29. 29.
    Zheng, Z., Lu, P., Lantz, B.: Commercial truck crash injury severity analysis using gradient boosting data mining model. J. Saf. Res. 65, 115–124 (2018)CrossRefGoogle Scholar
  30. 30.
    Delen, D., Sharda, R., Bessonov, M.: Identifying significant predictors of injury severity in traffic accidents using a series of artificial neural networks. Accid. Anal. Prev. 38(3), 434–444 (2006)CrossRefGoogle Scholar
  31. 31.
    Chang, L.-Y., Wang, H.-W.: Analysis of traffic injury severity: an application of non-parametric classification tree techniques. Accid. Anal. Prev. 38(5), 1019–1027 (2006)CrossRefGoogle Scholar
  32. 32.
    Nguyen, C.H., Cai, Chen, F.: Automatic classification of traffic incident’s severity using machine learning approaches. IET Intel. Transp. Syst. 11, 615–623 (2017)CrossRefGoogle Scholar
  33. 33.
    Kheradpisheh, S., Sharifizadeh, F., Nowzari-Dalini, A., Ganjtabesh, M., Ebrahimpour, R.: Mixture of feature specified experts. Inf. Fusion 20, 242–251 (2014)CrossRefGoogle Scholar
  34. 34.
    Li, L., Zou, B., Hu, Q., Wu, X., Yu, D.: Dynamic classifier ensemble using classification confidence. Neurocomputing 99, 581–591 (2013)CrossRefGoogle Scholar
  35. 35.
    Yu, J.S., Chen, J., Xiang, Z.Q., Zou, Y.X.: A hybrid convolutional neural networks with extreme learning machine for WCE image classification. In: IEEE International Conference on Robotics and Biomimetics (ROBIO) (2015)Google Scholar
  36. 36.
    McDonnell, M.D., Tissera, M.D., Vladusich, T., Van Schaik, A., Tapson, J.: Fast, simple and accurate handwritten digit classification by training shallow neural network classifiers with the ‘extreme learning machine’algorithm. PLoS One 10(8), e0134254 (2015)CrossRefGoogle Scholar
  37. 37.
    Liu, Y., Yao, X.: Ensemble learning via negative correlation. Neural Netw. 12(10), 1399–1404 (1999)CrossRefGoogle Scholar
  38. 38.
    Masoudnia, S., Ebrahimpour, R., Arani, S.: Incorporation of a regularization term to control negative correlation in mixture of experts. Neural Process. Lett. 36(1), 31–47 (2012)CrossRefGoogle Scholar
  39. 39.
    Islam, M., Yao, X., Nirjon, S., Islam, M., Murase, K.: Bagging and boosting negatively correlated neural networks. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 38(3), 771–784 (2008)CrossRefGoogle Scholar
  40. 40.
    Liu, Y., Yao, X.: Simultaneous training of negatively correlated neural networks in an ensemble. IEEE Trans.Syst. Man Cybern. Part B (Cybern.) 29(6), 716–725 (1999)CrossRefGoogle Scholar
  41. 41.
    Ebrahimpour, R., Sadeghnejad, N., Masoudnia, S., Arani, S.: Boosted pre-loaded mixture of experts for low-resolution face recognition. Int. J. Hybrid Intell. Syst. 9(3), 145–158 (2012)CrossRefGoogle Scholar
  42. 42.
    Lotfi, M., Motamedi, S., Sharifian, S.: Time-based feedback-control framework for real-time video surveillance systems with utilization control. J. Real-Time Image Proc. (2016). Google Scholar
  43. 43.
    Zarándy, Á, Nemeth, M., Nagy, Z., Kiss, A., Santha, L., Zsedrovits: A real-time multi-camera vision system for UAV collision warning and navigation. J. Real-Time Image Proc. 4, 709–724 (2016)CrossRefGoogle Scholar
  44. 44.
    Puri, A.: A survey of unmanned aerial vehicles (UAV) for traffic surveillance. Department of computer science and engineering, University of South Florida (2005)Google Scholar
  45. 45.
    Pearson, K.: Liii. On lines and planes of closest fit to systems of points in space. Lond. Edinb. Dublin Philos. Mag. J. Sci. 2(11), 559–572 (1901)CrossRefzbMATHGoogle Scholar
  46. 46.
    Hall, M.: Correlation-based feature selection for machine learning. PhD thesis, Department of Computer Science, University of Waikato Hamilton (1999)Google Scholar
  47. 47.
    Hamon, J.: Optimisation Combinatoire Pour La Sélection De Variables En Régression En Grande Dimension: Application En Génétique Animale. Université des Sciences et Technologie de Lille-Lille I. (2013)Google Scholar
  48. 48.
    Mofarreh-Bonab, M., Mofarreh-Bonab, M.: Color image compression using PCA. Int. J. Comput. Appl. 111(5):16–19 (2015)Google Scholar
  49. 49.
    Zhang, L., Yang, F., Zhang, Y.D., Zhu, Y.J.: Road crack detection using deep convolutional neural network. In: International Conference on Image Processing (ICIP), pp. 3708–3712 (2016)Google Scholar
  50. 50.
    Weng, Q., Mao, Z., Lin, J., Liao, X.: Land-use scene classification based on a cnn using a constrained extreme learning machine. Int. J. Remote Sens. pp. 1–19 (2018)Google Scholar
  51. 51.
    Martinel, N., Piciarelli, C., Foresti, G., Micheloni C.: Mobile food recognition with an extreme deep tree. In: Proceedings of the 10th International Conference on Distributed Smart Camera (2016)Google Scholar
  52. 52.
    Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)CrossRefGoogle Scholar
  53. 53.
    Abbasi, E., Shiri, M., Ghatee, M.: A regularized root–quartic mixture of experts for complex classification problems. Knowl.-Based Syst. 110, 98–109 (2016)CrossRefGoogle Scholar
  54. 54.
    Huang, G.-B., Zhu, Q.-Y., Siew, C.-K.: Extreme learning machine: a new learning scheme of feedforward neural networks. In: International Joint Conference on Neural Networks (2004)Google Scholar
  55. 55.
    Huang, G.-B., Zhou, H., Ding, X., Zhang, R.: Extreme learning machine for regression and multiclass classification. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 42(2), 513–529 (2012)CrossRefGoogle Scholar
  56. 56.
    Huang, G.-B., Liang, N.-Y., Rong, H.-J., Saratchandran, P., Sundararajan N.: “On-line sequential extreme learning machine. Comput. Intell. 2005, 232–237 (2005)Google Scholar
  57. 57.
    Zhu, W., Miao, J., Qing, L.: Constrained extreme learning machine: a novel highly discriminative random feedforward neural network. In: International Joint Conference on Neural Networks (IJCNN) (2014)Google Scholar
  58. 58.
    Tian, H.X., Mao, Z.Z.: An ensemble ELM based on modified AdaBoost. RT algorithm for predicting the temperature of molten steel in ladle furnace. IEEE Trans. Autom. Sci. Eng. 7(1), 73–80 (2010)CrossRefGoogle Scholar
  59. 59.
    Huang, Y., Suen, C.: The behavior-knowledge space method for combination of multiple classifiers. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (1993)Google Scholar
  60. 60.
    Kuncheva, L., Bezdek, J., Duin, R.: Decision templates for multiple classifier fusion: an experimental comparison. Pattern Recognit. 34(2), 299–314 (2001)CrossRefzbMATHGoogle Scholar
  61. 61.
    Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. John Wiley & Sons (2004)Google Scholar
  62. 62.
    Pashaei, A., Ghatee, M., Sajedi, H.: Accident images analysis dataset. Amirkabir University of Technology, 2018. (Online). Accessed 2018
  63. 63.
    Yann, L., Corinna, C., Christopher, J.: The Mnist Database of Handwritten Digits (Online). (1998)
  64. 64.
    Vahdatpour, M., Sajedi, H., Ramezani, F.: Air pollution forecasting from sky images with shallow and deep classifiers. Earth Sci. Inf. 11(3), 413–422 (2018)CrossRefGoogle Scholar
  65. 65.
    Cheng, J., Huang, W., Cao, S., Yang, R., Yang, W., Yun, Z., Wang, Z., Feng, Q.: Enhanced performance of brain tumor classification via tumor region augmentation and partition. PLoS One 10(10), 0140381 (2015)Google Scholar
  66. 66.
    Arróspide, J., Salgado, L., Nieto, M.: Video analysis based vehicle detection and tracking using an MCMC sampling framework. EURASIP J. Adv. Signal Process. (2012)Google Scholar
  67. 67.
    Wan, L., Zeiler, M., Zhang, S., Le Cun, Y., Fergus R.: Regularization of neural networks using dropconnect. In: International Conference on Machine Learning (2013)Google Scholar
  68. 68.
    Haut, J., Paoletti, M., Plaza, J., Plaza, A.: Fast dimensionality reduction and classification of hyperspectral images with extreme learning machines. J. Real-Time Image Proc. 15(3), 439–462 (2018)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Computer ScienceAmirkabir University of TechnologyTehranIran
  2. 2.School of Mathematics, Statistics and Computer Science, College of ScienceUniversity of TehranTehranIran

Personalised recommendations