Skip to main content

Recognition and location of steel structure surface corrosion based on unmanned aerial vehicle images

Abstract

Corrosion of steel structures may cause failures that can adversely affect building performance. A flexible and convenient monitoring method that utilizes unmanned aerial vehicles (UAVs) can detect the corrosion areas on structures in time to avoid potential safety accidents. As an attempt to utilize UAV monitoring, this paper proposes an image detection approach for corrosion, which includes three steps: super-pixel segmentation, neural network identification and corrosion area location. First, the UAV images are presegmented by the simple linear iterative clustering (SLIC) super-pixel segmentation algorithm, and the segmentation parameters are optimized according to the corrosion segmentation ratio to evaluate the feasibility of the proposed method. Then, a corrosion segmentation dataset is established, and the feature pyramid network (FPN) and path aggregation network (PANet) algorithms are trained to obtain a recognition model with a recognition accuracy of 97.15%. Combining SLIC super-pixel segmentation labels with UAV flight records, a two-stage corrosion location method for overall monitoring and local monitoring of structures is proposed. The method is evaluated on the case of the Tianjin Jinbu Bridge, and the error of this method is approximately 4%. The results demonstrate that the two-stage monitoring method can obtain the proportion of the corrosion area and the relative structure position. The proposed approach can facilitate the early and comprehensive detection of corrosion along the entire route of a steel structure via integration with UAVs.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

References

  1. 1.

    ASTM D610-08 (2019) Standard practice for evaluating degree of rusting on painted steel surfaces. ASTM International, West Conshohocken.https://doi.org/10.1520/d0610-08

  2. 2.

    Hou B, Li X, Ma X et al (2017) The cost of corrosion in China. Mater Degrad 1(1):1–10. https://doi.org/10.1038/s41529-017-0005-2

    Article  Google Scholar 

  3. 3.

    Agdas D, Rice JA, Martinez JR et al (2016) Comparison of visual inspection and structural-health monitoring as bridge condition assessment methods. J Perform Constr Facil 30(3):04015049. https://doi.org/10.1061/(asce)cf.1943-5509.0000802

    Article  Google Scholar 

  4. 4.

    LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444. https://doi.org/10.1038/nature14539

    Article  Google Scholar 

  5. 5.

    Ruan S, Tang C, Xu Z, Jin Z et al (2020) Multi-pose face recognition based on deep learning in unconstrained scene. Appl Sci 10(13):4669. https://doi.org/10.3390/app10134669

    Article  Google Scholar 

  6. 6.

    Kim W, Jang HY, Ko Y et al (2020) Inconsistency in the use of the term “validation” in studies reporting the performance of deep learning algorithms in providing diagnosis from medical imaging. PLoS ONE 15(9):e0238908. https://doi.org/10.1371/journal.pone.0238908

    Article  Google Scholar 

  7. 7.

    Yamashkin SA, Yamashkin AA, Zanozin VV et al (2020) Improving the efficiency of deep learning methods in remote sensing data analysis: geosystem approach. IEEE Access 8:179516–179529. https://doi.org/10.1109/ACCESS.2020.3028030

    Article  Google Scholar 

  8. 8.

    Cha YJ, Choi W, Oral B (2017) Deep learning-based crack damage detection using convolutional neural networks. Comput Aided Civil Infrastruct Eng 32(5):361–378. https://doi.org/10.1111/mice.12263

    Article  Google Scholar 

  9. 9.

    Kang D, Cha YJ (2018) Autonomous UAVs for structural health monitoring using deep learning and an ultrasonic beacon system with geo-tagging. Comput Aided Civil Infrastruct Eng 33(10):885–902. https://doi.org/10.1111/mice.12375

    Article  Google Scholar 

  10. 10.

    Dorafshan S, Robert JT, Marc M (2018) Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete. Constr Build Mater 186:1031–1045. https://doi.org/10.1016/j.conbuildmat.2018.08.011

    Article  Google Scholar 

  11. 11.

    Kohiyama M, Oka K, Yamashita T (2020) Detection method of unlearned pattern using support vector machine in damage classification based on deep neural network. Struct Control Health Monit 27(8):e2552. https://doi.org/10.1002/stc.2552

    Article  Google Scholar 

  12. 12.

    MoralesValdez J, LopezPacheco M, Yu W (2020) Automated damage location for building structures using the hysteretic model and frequency domain neural networks. Struct Control Health Monit 27(9):e2584. https://doi.org/10.1002/stc.2584

    Article  Google Scholar 

  13. 13.

    Khayatazad M, De PL, De WW (2020) Detection of corrosion on steel structures using automated image processing. Dev Built Environ 3:100022. https://doi.org/10.1016/j.dibe.2020.100022

    Article  Google Scholar 

  14. 14.

    Ortiz A, Francisco BP, Emilio GF (2016) Vision-based corrosion detection assisted by a micro-aerial vehicle in a vessel inspection application. Sensors 16(12):2118. https://doi.org/10.3390/s16122118

    Article  Google Scholar 

  15. 15.

    Xu J, Gui CQ, Han QH (2020) Recognition of rust grade and rust ratio of steel structures based on ensembled convolutional neural network. Comput Aided Civil Infrastruct Eng 35(10):1160–1174. https://doi.org/10.1111/mice.12563

    Article  Google Scholar 

  16. 16.

    Li Z, Liu Y, Walker R et al (2010) Towards automatic power line detection for a UAV surveillance system using pulse coupled neural filter and an improved Hough transform. Mach Vis Appl 21(5):677–686. https://doi.org/10.1007/s00138-009-0206-y

    Article  Google Scholar 

  17. 17.

    Hausamann D, Zirnig W, Schreier G et al (2005) Monitoring of gas pipelines-a civil UAV application. Aircr Eng Aerosp Technol 77(5):352–360. https://doi.org/10.1108/00022660510617077

    Article  Google Scholar 

  18. 18.

    Huang C, Wei HL, Rau JY et al (2019) Use of principal components of UAV-acquired narrow-band multispectral imagery to map the diverse low stature vegetation fAPAR. GISci Remote Sens 56(4):605–623. https://doi.org/10.1080/15481603.2018.1550873

    Article  Google Scholar 

  19. 19.

    Michez A, Piégay H, Lisein J et al (2016) Classification of riparian forest species and health condition using multi-temporal and hyperspatial imagery from unmanned aerial system. Environ Monit Assess 188(3):146. https://doi.org/10.1007/s10661-015-4996-2

    Article  Google Scholar 

  20. 20.

    De CG, Balta H, Doroftei D et al (2014) UAS deployment and data processing during the Balkans flooding. In: 2014 IEEE international symposium on safety, security, and rescue robotics. IEEE, pp 1–4. https://doi.org/10.1109/ssrr.2014.7017670

  21. 21.

    Rathinam S, Kim ZW, Sengupta R (2008) Vision-based monitoring of locally linear structures using an unmanned aerial vehicle. J Infrastruct Syst 14(1):52–63. https://doi.org/10.1061/(asce)1076-0342(2008)14:1(52)

    Article  Google Scholar 

  22. 22.

    Oczipka M, Bemmann J, Piezonka H et al (2009) Small drones for geo-archaeology in the steppe: locating and documenting the archaeological heritage of the Orkhon Valley in Mongolia. In: Remote sensing for environmental monitoring, GIS applications, and geology IX. International Society for Optics and Photonics, vol 7478, p 747806.https://doi.org/10.1117/12.830404

  23. 23.

    Zhang CS, Ahmed E (2012) An unmanned aerial vehicle-based imaging system for 3D measurement of unpaved road surface distresses. Comput Aided Civil Infrastruct Eng 27(2):118–129. https://doi.org/10.1111/j.1467-8667.2011.00727.x

    Article  Google Scholar 

  24. 24.

    Yoon H, Jaeho S, Billie FS (2018) Structural displacement measurement using an unmanned aerial system. Comput Aided Civil Infrastruct Eng 33(3):183–192. https://doi.org/10.1111/mice.12338

    Article  Google Scholar 

  25. 25.

    Reagan D, Alessandro S, Christopher N (2018) Feasibility of using digital image correlation for unmanned aerial vehicle structural health monitoring of bridges. Struct Health Monit 17(5):1056–1072. https://doi.org/10.1177/1475921717735326

    Article  Google Scholar 

  26. 26.

    Messinger M, Silman M (2016) Unmanned aerial vehicles for the assessment and monitoring of environmental contamination: an example from coal ash spills. Environ Pollut 218:889–894. https://doi.org/10.1016/j.envpol.2016.08.019

    Article  Google Scholar 

  27. 27.

    Hackl J, Adey BT, Woźniak M et al (2017) Use of unmanned aerial vehicle photogrammetry to obtain topographical information to improve bridge risk assessment. J Infrastruct Syst 24(1):04017041. https://doi.org/10.1061/(asce)is.1943-555x.0000393

    Article  Google Scholar 

  28. 28.

    Inzerillo L, Gaetano DM, Ronald R (2018) Image-based 3D reconstruction using traditional and UAV datasets for analysis of road pavement distress. Autom Constr 96:457–469. https://doi.org/10.1016/j.autcon.2018.10.010

    Article  Google Scholar 

  29. 29.

    Ellenberg A, Kontsos A, Moon F et al (2016) Bridge Deck delamination identification from unmanned aerial vehicle infrared imagery. Autom Constr 72:155–165. https://doi.org/10.1016/j.autcon.2016.08.024

    Article  Google Scholar 

  30. 30.

    Omar T, Moncef LN (2017) Remote sensing of concrete bridge decks using unmanned aerial vehicle infrared thermography. Autom Constr 83:360–371. https://doi.org/10.1016/j.autcon.2017.06.024

    Article  Google Scholar 

  31. 31.

    Choi J, Yeum C, Dyke S et al (2018) Computer-aided approach for rapid post-event visual evaluation of a building façade. Sensors 18(9):3017. https://doi.org/10.3390/s18093017

    Article  Google Scholar 

  32. 32.

    Ellenberg A, Branco L, Krick A et al (2014) Use of unmanned aerial vehicle for quantitative infrastructure evaluation. J Infrastruct Syst 21(3):04014054. https://doi.org/10.1061/(asce)is.1943-555x.0000246

    Article  Google Scholar 

  33. 33.

    Ong WH, Chiu W, Kuen T et al (2017) Determination of the state of strain of large floating covers using unmanned aerial vehicle (UAV) aided photogrammetry. Sensors 17(8):1731. https://doi.org/10.3390/s17081731

    Article  Google Scholar 

  34. 34.

    Ren X, Malik J (2003) Learning a classification model for segmentation. In: Proc of the 9th IEEE international conference on computer vision. IEEE Computer Society, Washington DC, pp 10–17. https://doi.org/10.1109/iccv.2003.1238308

  35. 35.

    Shi J, Malik J (2000) Normalized cuts and image segmentation. IEEE Trans Pattern Anal Mach Intell 22(8):888–905. https://doi.org/10.1109/34.868688

    Article  Google Scholar 

  36. 36.

    Felzenszwalb PF, Huttenlocher DP (2004) Efficient graph-based image segmentation. Int J Comput Vis 59(2):167–181. https://doi.org/10.1023/B:VISI.0000022288.19776.77

    Article  Google Scholar 

  37. 37.

    Vincent L, Soille P (1991) Watersheds in digital spaces: an efficient algorithm based on immersion simulations. IEEE Trans Pattern Anal Mach Intell 13(6):583–598. https://doi.org/10.1109/34.87344

    Article  Google Scholar 

  38. 38.

    Alex L, Adrian S, Kutulakos KN et al (2009) Turbopixels: fast superpixels using geometric flows. IEEE Trans Pattern Anal Mach Intell 31(12):2290–2297. https://doi.org/10.1109/TPAMI.2009.96

    Article  Google Scholar 

  39. 39.

    Achanta R, Shaji A, Smith K et al (2010) SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell 34(11):2274–2282. https://doi.org/10.1109/TPAMI.2012.120

    Article  Google Scholar 

  40. 40.

    Borsos B, Nagy L, Iclănzan D et al (2019) Automatic detection of hard and soft exudates from retinal fundus images. Acta Univers Sapientiae Inform 11(1):65–79. https://doi.org/10.2478/ausi-2019-0005

    Article  MATH  Google Scholar 

  41. 41.

    Chen J, Liu D (2021) Bottom-up image detection of water channel slope damages based on superpixel segmentation and support vector machine. Adv Eng Inform 47:101205. https://doi.org/10.1016/j.aei.2020.101205

    Article  Google Scholar 

  42. 42.

    Huang Q, Huang Y, Luo Y et al (2020) Segmentation of breast ultrasound image with semantic classification of superpixels. Med Image Anal 61:101657. https://doi.org/10.1016/j.media.2020.101657

    Article  Google Scholar 

  43. 43.

    Lin TY, Dollár P, Girshick R et al (2017) Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2117–2125. https://doi.org/10.1109/cvpr.2017.106

  44. 44.

    Liu S, Qi L, Qin H et al (2018) Path aggregation network for instance segmentation. IN: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8759–8768. https://doi.org/10.1109/cvpr.2018.00913

  45. 45.

    Russell BC, Torralba A, Murphy KP et al (2008) LabelMe: a database and web-based tool for image annotation. Int J Comput Vis 77(1–3):157–173. https://doi.org/10.1007/s11263-007-0090-8

    Article  Google Scholar 

Download references

Acknowledgements

The authors express their appreciation for financial support from the Specialized Research Fund of the Nature Science Foundation of China (No. 51525803).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jie Xu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Han, Q., Zhao, N. & Xu, J. Recognition and location of steel structure surface corrosion based on unmanned aerial vehicle images. J Civil Struct Health Monit 11, 1375–1392 (2021). https://doi.org/10.1007/s13349-021-00515-7

Download citation

Keywords

  • Corrosion recognition
  • Location
  • UAV image
  • SLIC super-pixel segmentation