Skip to main content
SpringerLink
Log in

A deep learning approach to automatic road surface monitoring and pothole detection

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

Anomalies in road surface not only impact road quality but also affect driver safety, mechanic structure of the vehicles, and fuel consumption. Several approaches have been proposed to automatic monitoring of the road surface condition in order to assess road roughness and to detect potholes. Some of these approaches adopt a crowdsensing perspective by using a built-in smartphone accelerometer to sense the road surface. Although the crowdsensing perspective has several advantages as ubiquitousness and low cost, it has certain sensibility to the false positives produced by man-made structures, driver actions, and road surface characteristics that cannot be considered as road anomalies. For this reason, we propose a deep learning approach that allows us (a) to automatically identify the different kinds of road surface, and (b) to automatically distinguish potholes from destabilizations produced by speed bumps or driver actions in the crowdsensing-based application context. In particular, we analyze and apply different deep learning models: convolutional neural networks, LSTM networks, and reservoir computing models. The experiments were carried out with real-world information, and the results showed a promising accuracy in solving both problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. ASTM E867 - 06 (2012) standard terminology relating to Vehicle-Pavement systems. https://www.astm.org/Standards/E867.htm. Accessed: 2018-10-23

  2. Alessandroni G, Carini A, Lattanzi E, Freschi V, Bogliolo A (2017) A study on the influence of speed on road roughness sensing The SmartRoadSense case. Sensors 17(2):1–20

  3. Alvarez JR, Arroqui M, Mangudo P, Toloza J, Jatip D, Rodriguez JM, Teyseyre A, Sanz C, Zunino A, Machado C, Mateos C (2018) Body condition estimation on cows from depth images using convolutional neural networks. Comput Electron Agric 155:12–22

    Article  Google Scholar 

  4. Beckman GH, Polyzois D, Cha Y-J (2019) Deep learning-based automatic volumetric damage quantification using depth camera. Autom Constr 99:114–124

    Article  Google Scholar 

  5. Bello-Salau H, Aibinu AM, Onumanyi AJ, Onwuka EN, Dukiya JJ, Ohize H (2018) New road anomaly detection and characterization algorithm for autonomous vehicles. Applied Computing and Informatics

  6. Bianchi FM, De Santis E, Rizzi A, Sadeghian A (2015) Short-term electric load forecasting using echo state networks and pca decomposition. IEEE Access 3:1931–1943

    Article  Google Scholar 

  7. Bianchi FM, Livi L, Alippi C (2018) Investigating echo-state networks dynamics by means of recurrence analysis. IEEE Trans Neural Netw Learn Syst 29(2):427–439

    Article  MathSciNet  Google Scholar 

  8. Bianchi FM, Scardapane S, Løkse S, Jenssen R (2018) Reservoir computing approaches for representation and classification of multivariate time series. arXiv:1803.07870

  9. Bianchi FM, Scardapane S, Uncini A, Rizzi A, Sadeghian A (2015) Prediction of telephone calls load using echo state network with exogenous variables. Neural Netw 71:204–213

    Article  Google Scholar 

  10. Carrera F, Guerin, S, Thorp JB (2013) By the people, for the people: the crowdsourcing of “Streetbump”: an automatic pothole mapping app. ISPRS - Int Arch Photogramm Remote Sens Spat Inf Sci XL-4/W1:19–23

    Article  Google Scholar 

  11. Chollet F (2017) Deep Learning with Python, 1st edn. Manning Publications Co., Greenwich

    Google Scholar 

  12. Douangphachanh V, Oneyama H (2013) Estimation of road roughness condition from smartphones under realistic settings. In: 2013 13th International Conference on ITS Telecommunications (ITST)

  13. Edy SP, Vesilo R (2017) Window-size impact on detection rate of wearable-sensor-based fall detection using supervised machine learning. In: 2017 IEEE life sciences conference (LSC). IEEE, pp 21–26

  14. Ghanta S, Shamsabadi SS, Dy J, Wang M, Birken R (2015) A hessian-based methodology for automatic surface crack detection and classification from pavement images. In: Structural health monitoring and inspection of advanced materials, Aerospace, and Civil Infrastructure 2015

  15. Gorges C, Öztürk K, Liebich R (2019) Impact detection using a machine learning approach and experimental road roughness classification. Mech Syst Signal Process 117:738–756

    Article  Google Scholar 

  16. Harikrishnan H, Gopi VP (2017) Vehicle vibration signal processing for road surface monitoring. IEEE Sens. J. 17(16):5192–5197

    Article  Google Scholar 

  17. Ibañez R, Soria A, Teyseyre AR, Berdun L, Campo M (2016) A comparative study of machine learning techniques for gesture recognition using kinect. In: Handbook of Research on Human-Computer Interfaces, Developments, and Applications. IGI Global, pp 1–22

  18. Ibaṅez R, Soria Á, Teyseyre AR, Campo M (2014) Easy gesture recognition for kinect. Adv Eng Softw 76:171–180

    Article  Google Scholar 

  19. Jain AK (2010) Award winning papers from the 19th International Conference on Pattern Recognition (ICPR). Pattern Recogn Lett 31(8):651 – 666

    Article  Google Scholar 

  20. Kadous MW, Sammut C (2005) Classification of multivariate time series and structured data using constructive induction. Mach Learn 58(2-3):179–216

    Article  Google Scholar 

  21. Kertesz I, Lovas T, Barsi A (2007) Measurement of road roughness by low-cost photogrammetric system, pp 36

  22. Koch C, Brilakis I (2011) Pothole detection in asphalt pavement images. Adv Eng Inform 25(3):507–515

    Article  Google Scholar 

  23. Kumar P, Angelats E (2017) An automated road roughness detection from mobile laser scanning data. ISPRS - International Archives of the Photogrammetry. Remote Sens Spatial Inf Sci XLII-1/W1:91–96

    Google Scholar 

  24. Lepine J, Rouillard V, Sek M (2018) Evaluation of machine learning algorithms for detection of road induced shocks buried in vehicle vibration signals. Proc. Inst. Mech. Eng. Pt. D J. Automobile Eng., pp 095440701875620

  25. Li D, Han M, Wang J (2012) Chaotic time series prediction based on a novel robust echo state network. IEEE Trans Neural Netw Learn Syst 23(5):787–799

    Article  Google Scholar 

  26. Li Q, Yao M, Yao X, Xu B (2009) A real-time 3D scanning system for pavement distortion inspection. Measur Sci Technol 21(1):015702

    Article  Google Scholar 

  27. Li X, Goldberg DW (2018) Toward a mobile crowdsensing system for road surface assessment. Comput Environ Urban Syst 69:51–62

    Article  Google Scholar 

  28. Lukoševičius M, Jaeger H (2009) Reservoir computing approaches to recurrent neural network training. Comput Sci Rev 3(3):127–149

    Article  Google Scholar 

  29. Michie D, Spiegelhalter DJ, Taylor CC (1994) Machine Learning, Neural and statistical classification. Ellis Horwood

  30. Mohamed A, Fouad MMM, Elhariri E, El-Bendary N, Zawbaa HM, Tahoun M, Hassanien AE (2015) RoadMonitor: An intelligent road surface condition monitoring system. In: Advances in Intelligent Systems and Computing, pp 377–387

  31. Monteserin A (2018) Potholes vs. speed bumps: A multivariate time series classification approach. In: Proceedings of the 1st Workshop on Knowledge Discovery and User Modelling for Smart Cities co-located with 24th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, UMCit@KDD 2018, London, pp 36–40

  32. Ngwangwa HM, Heyns PS (2014) Application of an ANN-based methodology for road surface condition identification on mining vehicles and roads. J Terramech 53:59–74

    Article  Google Scholar 

  33. Perttunen M, Mazhelis O, Cong F, Kauppila M, Leppänen T, Kantola J, Collin J, Pirttikangas S, Haverinen J, Ristaniemi T, Riekki J (2011) Distributed road surface condition monitoring using mobile phones. In: Lecture Notes in Computer Science, pp 64–78

  34. Rodan A, Sheta AF, Faris H (2017) Bidirectional reservoir networks trained using svm+ privileged information for manufacturing process modeling. Soft Comput 21(22):6811–6824

    Article  Google Scholar 

  35. Russell S, Norvig P (2009) Artificial Intelligence: A Modern Approach, 3rd edn. Prentice Hall Press, Upper Saddle River

    MATH  Google Scholar 

  36. Md S, Mandal S, Islam S (2015) A support vector method for automated road anomaly detection using mobile device. Int J Comput Appl Technol 127(9):16–19

    Google Scholar 

  37. Sainath T, Vinyals O, Senior A, Sak H (2015) Convolutional, long short-term memory, fully connected deep neural networks. In: ICASSP

  38. Sainath TN, Kingsbury B, Saon G, Soltau H, Mohamed A-R, Dahl G, Ramabhadran B (2015) Deep convolutional neural networks for large-scale speech tasks. Neural Netw 64(C):39–48

    Article  Google Scholar 

  39. Sak H, Senior AW, Beaufays F (2014) Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition. arXiv:1402.1128

  40. Salvador S, Chan P (2007) Toward accurate dynamic time warping in linear time and space. Intell Data Anal 11(5):561–580

    Article  Google Scholar 

  41. Schrauwen B, Verstraeten D, Van Campenhout J (2007) An overview of reservoir computing: theory, applications and implementations. In: Proceedings of the 15th European Symposium on Artificial Neural Networks. pp 471–482

  42. Silva N, Shah V, Soares J, Rodrigues H (2018) Road anomalies detection system evaluation. Sensors 18(7):1–20

  43. Tai Y-C, Chan C-W, Hsu JY-J (2010) Automatic road anomaly detection using smart mobile device. In: 2010 15th Conference on Artificial Intelligence and Applications (TAAI)

  44. Trentin E, Scherer S, Schwenker F (2015) Emotion recognition from speech signals via a probabilistic echo-state network. Pattern Recogn Lett 66:4–12

    Article  Google Scholar 

  45. Wang J, Chen Y, Hao S, Peng X, Hu L (2017) Deep learning for sensor-based activity recognition: A survey. arXiv:1707.03502

  46. Xing Z, Pei J, Keogh E (2010) A brief survey on sequence classification. SIGKDD Explor Newsl 12(1):40–48

    Article  Google Scholar 

  47. Yaqub R, Burrell J, Crutcher X, Morrow J (2017) Prototype model of potholes and road-roughness detection and reporting system. In: 2017 IEEE International Conference on Smart Energy Grid Engineering (SEGE)

  48. Zeng H, Park H, Smith BL, Parkany E (2017) Feasibility assessment of a Smartphone-Based application to estimate road roughness. KSCE J Civ Eng 22(8):3120–3129

    Article  Google Scholar 

  49. Zhang Z, Deng F, Huang Y, Bridgelall R (2015) Road roughness evaluation using in-pavement strain sensors. Smart Mater Struct 24(11):115029

    Article  Google Scholar 

  50. Zhang Z, Sun C, Bridgelall R, Sun M (2018) Application of a machine learning method to evaluate road roughness from connected vehicles. J Transp Eng Part B: Pavements 144(4):04018043

    Article  Google Scholar 

Download references

Acknowledgments

We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.

Author information

Authors and Affiliations

  1. ISISTAN Research Institute (CONICET-UNICEN), Tandil, Argentina

    Braian Varona, Ariel Monteserin & Alfredo Teyseyre

Authors
  1. Braian Varona
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ariel Monteserin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alfredo Teyseyre
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ariel Monteserin.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Varona, B., Monteserin, A. & Teyseyre, A. A deep learning approach to automatic road surface monitoring and pothole detection. Pers Ubiquit Comput 24, 519–534 (2020). https://doi.org/10.1007/s00779-019-01234-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-019-01234-z

Keywords

Search

Navigation