MaDnet: multi-task semantic segmentation of multiple types of structural materials and damage in images of civil infrastructure

Abstract

Manual visual inspection is the most common means of assessing the condition of civil infrastructure in the United States, but can be exceedingly laborious, time-consuming, and dangerous. Research has focused on automating parts of the inspection process using unmanned aerial vehicles for image acquisition, followed by deep learning techniques for damage identification. Existing deep learning methods and datasets for inspections have typically been developed for a single damage type. However, most guidelines for inspections require the identification of multiple damage types and describe evaluating the significance of the damage based on the associated material type. Thus, the identification of material type is important in understanding the meaning of the identified damage. Training separate networks for the tasks of material and damage identification fails to incorporate this intrinsic interdependence between them. We hypothesize that a network that incorporates such interdependence directly will have a better accuracy in material and damage identification. To this end, a deep neural network, termed the material-and-damage-network (MaDnet), is proposed to simultaneously identify material type (concrete, steel, asphalt), as well as fine (cracks, exposed rebar) and coarse (spalling, corrosion) structural damage. In this approach, semantic segmentation (i.e., assignment of each pixel in the image with a material and damage label) is employed, where the interdependence between material and damage is incorporated through shared filters learned through multi-objective optimization. A new dataset with pixel-level labels identifying the material and damage type is developed and made available to the research community. Finally, the dataset is used to evaluate MaDnet and demonstrate the improvement in pixel accuracy over employing independent networks.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

References

  1. 1.

    Koch C, Georgieva K, Kasireddy V et al (2015) A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure. Adv Eng Inf 29:196–210

    Article  Google Scholar 

  2. 2.

    National Transportation Safety Board (NTSB) (2007) Collapse of I-35W highway bridge Minneapolis, Minnesota August 1, 2007. Highw Accid Rep 178:86

    Google Scholar 

  3. 3.

    ATC (Applied Technology Council) (2005) Field manual: postearthquake safety evaluation of building. ATC-20-1, Redwood City, CA

  4. 4.

    Marroquin A (2017) Inspections bring high degree of difficulty at Hoover Dam bypass bridge. In: Las Vegas Rev. https://www.reviewjournal.com/local/local-nevada/inspections-bring-high-degree-of-difficulty-at-hoover-dam-bypass-bridge/. Accessed 23 Aug 2017

  5. 5.

    Abdel-Qader I, Abudayyeh O, Kelly ME (2003) Analysis of edge-detection techniques for crack identification in bridges. J Comput Civ Eng 17:255–263. https://doi.org/10.1061/(ASCE)0887-3801(2003)17:4(255)

    Article  Google Scholar 

  6. 6.

    Jahanshahi MR, Kelly JS, Masri SF, Sukhatme GS (2009) A survey and evaluation of promising approaches for automatic image-based defect detection of bridge structures. Struct Infrastruct Eng 5:455–486. https://doi.org/10.1080/15732470801945930

    Article  Google Scholar 

  7. 7.

    Yamaguchi T, Hashimoto S (2010) Fast crack detection method for large-size concrete surface images using percolation-based image processing. Mach Vis Appl 21:797–809. https://doi.org/10.1007/s00138-009-0189-8

    Article  Google Scholar 

  8. 8.

    Nishikawa T, Yoshida J, Sugiyama T, Fujino Y (2012) Concrete crack detection by multiple sequential image filtering. Comput Civ Infrastruct Eng 27:29–47. https://doi.org/10.1111/j.1467-8667.2011.00716.x

    Article  Google Scholar 

  9. 9.

    Zhu Z, German S, Brilakis I (2011) Visual retrieval of concrete crack properties for automated post-earthquake structural safety evaluation. Autom Constr 20:874–883. https://doi.org/10.1016/j.autcon.2011.03.004

    Article  Google Scholar 

  10. 10.

    Paal SG, Jeon J-S, Brilakis I, DesRoches R (2015) Automated damage index estimation of reinforced concrete columns for post-earthquake evaluations. J Struct Eng 141:04014228. https://doi.org/10.1061/(ASCE)ST.1943-541X.0001200

    Article  Google Scholar 

  11. 11.

    Adhikari RS, Moselhi O, Bagchi A (2013) A study of image-based element condition index for bridge inspection. In: ISARC 2013—30th Int Symp Autom Robot Constr Mining, Held Conjunction with 23rd World Min Congr, pp 345–356

  12. 12.

    Chen PH, Shen HK, Lei CY, Chang LM (2012) Support-vector-machine-based method for automated steel bridge rust assessment. Autom Constr 23:9–19. https://doi.org/10.1016/j.autcon.2011.12.001

    Article  Google Scholar 

  13. 13.

    Bonnin-Pascual F, Ortiz A (2014) Corrosion detection for automated visual inspection. Dev Corros Prot. https://doi.org/10.5772/57209

    Article  Google Scholar 

  14. 14.

    Yeum CM, Dyke SJ (2015) Vision-based automated crack detection for bridge inspection. Comput Civ Infrastruct Eng 30:759–770. https://doi.org/10.1111/mice.12141

    Article  Google Scholar 

  15. 15.

    Hu Y, Zhao C (2010) A novel LBP based methods for pavement crack detection. J Pattern Recognit Res 5:140–147. https://doi.org/10.13176/11.167

    Article  Google Scholar 

  16. 16.

    Zhang W, Zhang Z, Qi D, Liu Y (2014) Automatic crack detection and classification method for subway tunnel safety monitoring. Sensors (Switzerl) 14:19307–19328. https://doi.org/10.3390/s141019307

    Article  Google Scholar 

  17. 17.

    Wu S, Zhong S, Liu Y (2017) Deep residual learning for image steganalysis. Multimed Tools Appl 2017:1–17

    Google Scholar 

  18. 18.

    Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. Int Conf Learn Represent. https://doi.org/10.1016/j.infsof.2008.09.005

    Article  Google Scholar 

  19. 19.

    LeCun Y, Bengio Y, Hinton G et al (2015) Deep learning. Nature 521:436–444. https://doi.org/10.1038/nature14539

    Article  Google Scholar 

  20. 20.

    Banga S, Gehani H, Bhilare S, et al (2018) 3D Topology Optimization using Convolutional Neural Networks. arXiv preprint arXiv:1808.07440v1

  21. 21.

    Lei X, Liu C, Du Z et al (2019) Machine learning-driven real-time topology optimization under moving morphable component-based framework. J Appl Mech Trans ASME. https://doi.org/10.1115/1.4041319

    Article  Google Scholar 

  22. 22.

    Sosnovik I, Oseledets I (2019) Neural networks for topology optimization. Russ J Numer Anal Math Model 34:215–223. https://doi.org/10.1515/rnam-2019-0018

    MathSciNet  Article  MATH  Google Scholar 

  23. 23.

    Hoskere V, Eick BA, Spencer BF, Smith MD, Foltz SD (2019) Deep Bayesian neural networks for damage quantification in miter gates of navigation locks. Struct Heal Monit. https://doi.org/10.1177/1475921719882086

    Article  Google Scholar 

  24. 24.

    Rafiei MH, Adeli H (2018) A novel unsupervised deep learning model for global and local health condition assessment of structures. Eng Struct 156:598–607. https://doi.org/10.1016/j.engstruct.2017.10.070

    Article  Google Scholar 

  25. 25.

    Bao Y, Tang Z, Li H, Zhang Y (2019) Computer vision and deep learning–based data anomaly detection method for structural health monitoring. Struct Heal Monit 18:401–421. https://doi.org/10.1177/1475921718757405

    Article  Google Scholar 

  26. 26.

    Ye XW, Jin T, Yun CB (2019) A review on deep learning—based structural health monitoring of civil infrastructures. Smart Struct Syst 24:567–586. https://doi.org/10.12989/sss.2019.24.5.567

    Article  Google Scholar 

  27. 27.

    Tsuchimoto K, Narazaki Y, Hoskere V, Spencer Jr. BF (2020) Rapid postearthquake safety evaluation of buildings using sparse acceleration measurements. Struct Heal Monit (in press)

  28. 28.

    Dorafshan S, Maguire M (2018) Bridge inspection: human performance, unmanned aerial systems and automation. J Civ Struct Heal Monit 8:443–476. https://doi.org/10.1007/s13349-018-0285-4

    Article  Google Scholar 

  29. 29.

    Chen F-C, Jahanshahi MR, Johnson D, Delp EJ (2019) Vision-based decision support for flood risk assessment using google street view images. Struct Heal Monit. https://doi.org/10.12783/SHM2019/32472

    Article  Google Scholar 

  30. 30.

    Alipour M, Harris DK (2020) A big data analytics strategy for scalable urban infrastructure condition assessment using semi-supervised multi-transform self-training. J Civ Struct Heal Monit 10:313–332. https://doi.org/10.1007/s13349-020-00386-4

    Article  Google Scholar 

  31. 31.

    Yeum CM, Dyke SJ, Ramirez J (2018) Visual data classification in post-event building reconnaissance. Eng Struct 155:16–24. https://doi.org/10.1016/J.ENGSTRUCT.2017.10.057

    Article  Google Scholar 

  32. 32.

    Dimitrov A, Golparvar-Fard M (2014) Vision-based material recognition for automated monitoring of construction progress and generating building information modeling from unordered site image collections. Adv Eng Informatics 28:37–49. https://doi.org/10.1016/j.aei.2013.11.002

    Article  Google Scholar 

  33. 33.

    Roberts D, Torres Calderon W, Tang S, Golparvar-Fard M (2020) Vision-based construction worker activity analysis informed by body posture. J Comput Civ Eng 34:04020017. https://doi.org/10.1061/(asce)cp.1943-5487.0000898

    Article  Google Scholar 

  34. 34.

    Fang Q, Li H, Luo X et al (2018) A deep learning-based method for detecting non-certified work on construction sites. Adv Eng Informatics 35:56–68. https://doi.org/10.1016/j.aei.2018.01.001

    Article  Google Scholar 

  35. 35.

    Kim H, Kim H, Hong YW, Byun H (2018) Detecting construction equipment using a region-based fully convolutional network and transfer learning. J Comput Civ Eng 32:04017082. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000731

    Article  Google Scholar 

  36. 36.

    Yeum CM, Choi J, Dyke SJ (2017) Autonomous image localization for visual inspection of civil infrastructure. Smart Mater Struct 26:035051. https://doi.org/10.1088/1361-665X/aa510e

    Article  Google Scholar 

  37. 37.

    Narazaki Y, Hoskere V, Hoang TA, Spencer Jr. BF (2017) Vision-based automated bridge component recognition integrated with high-level scene understanding. In: The 13th international workshop on advanced smart materials and smart structures technology (ANCRiSST), Tokyo, Japan

  38. 38.

    Narazaki Y, Hoskere V, Hoang TA et al (2019) Vision-based automated bridge component recognition with high-level scene consistency. Comput Civ Infrastruct Eng mice 2019:12505. https://doi.org/10.1111/mice.12505

    Article  Google Scholar 

  39. 39.

    Narazaki Y, Hoskere V, Hoang TA, Spencer Jr. BF (2018) Automated bridge component recognition using video data. In: The 7th world conference on structural control and monitoring, 7WCSCM

  40. 40.

    Zhang L, Yang F, Daniel Zhang Y, Zhu YJ (2016) Road crack detection using deep convolutional neural network. IEEE Int Conf Image Process 2016:3708–3712. https://doi.org/10.1109/ICIP.2016.7533052

    Article  Google Scholar 

  41. 41.

    Yeum CM (2016) Computer vision-based structural assessment exploiting large volumes of images. Theses Diss Available from ProQuest

  42. 42.

    Hoskere V, Narazaki Y, Hoang TA, Spencer BF (2017) Vision-based Structural Inspection using Multiscale Deep Convolutional Neural Networks. In: 3rd Huixian International Forum on Earthquake Engineering for Young Researchers, University of Illinois, Urbana-Champaign. Urbana, Il, USA

  43. 43.

    Cha YJ, Choi W, Suh G et al (2018) Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Comput Civ Infrastruct Eng 33:731–747. https://doi.org/10.1111/mice.12334

    Article  Google Scholar 

  44. 44.

    Rubio JJ, Kashiwa T, Laiteerapong T et al (2019) Multi-class structural damage segmentation using fully convolutional networks. Comput Ind 112:103121. https://doi.org/10.1016/j.compind.2019.08.002

    Article  Google Scholar 

  45. 45.

    Spencer BF, Hoskere V, Narazaki Y (2019) Advances in computer vision-based civil infrastructure inspection and monitoring. Engineering 5:199–222

    Article  Google Scholar 

  46. 46.

    FHWA (2004) National bridge inspection standards regulations (NBIS). Fed Regist 69:15–35

    Google Scholar 

  47. 47.

    Caruana R (1997) Multitask learning. Mach Learn 28:41–75. https://doi.org/10.1023/A:1007379606734

    Article  Google Scholar 

  48. 48.

    Kendall A, Gal Y, Cipolla R (2017) Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. Mach Learn. https://doi.org/10.1109/CVPR.2018.00781

    Article  Google Scholar 

  49. 49.

    Teichmann M, Weber M, Zöllner M et al (2018) MultiNet: real-time joint semantic reasoning for autonomous driving. In: IEEE intelligent vehicles symposium, proceedings. institute of electrical and electronics engineers Inc., pp 1013–1020

  50. 50.

    Moeskops P, Wolterink JM, van der Velden BHM et al (2016) Deep learning for multi-task medical image segmentation in multiple modalities. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics). Springer, pp 478–486

  51. 51.

    Facial Landmark Detection by Deep Multi-task Learning (2019) https://mmlab.ie.cuhk.edu.hk/projects/TCDCN.html. Accessed 15 Nov 2019

  52. 52.

    Shelhamer E, Long J, Darrell T et al (2015) Fully convolutional networks for semantic segmentation. IEEE Trans Pattern Anal Mach Intell 2015:640–651

    Google Scholar 

  53. 53.

    Krogh A, Hertz JA (1992) A simple weight decay can improve generalization. Adv Neural Inf Process Syst 4:950–957

    Google Scholar 

  54. 54.

    Kingma DP, Ba JL (2015) Adam: a method for stochastic optimization. Int Conf Learn Represent 2015:1–15. https://doi.org/10.1145/1830483.1830503

    Article  Google Scholar 

  55. 55.

    Abadi M, Barham P, Chen J et al (2016) TensorFlow: a system for large-scale machine learning. In: 12th symposium on operating systems design and implementation (2016). GA, pp 265–283

  56. 56.

    Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 580–587

  57. 57.

    Hoskere V, Narazaki Y, Hoang TA, Spencer Jr. BF (2017) Vision-based structural inspection using multiscale deep convolutional neural networks. In: 3rd Huixian International Forum on Earthquake Engineering for Young Researchers, University of Illinois, Urbana-Champaign

  58. 58.

    Chungwook S, Enrique V, Jhon PS et al (2016) Performance of low-rise reinforced concrete buildings in the 2016 Ecuador earthquake. https://datacenterhub.org/resources/14160. Accessed 6 Aug 2017

  59. 59.

    Bridgehunter.com: Historic Bridges of the United States (2017) https://bridgehunter.com/. Accessed 6 Aug 2017

  60. 60.

    US Army Corps of Engineers (2017) https://www.usace.army.mil/. Accessed 6 Aug 2017

  61. 61.

    Farabet C, Couprie C, Najman L, Lecun Y (2013) Learning hierarchical features for scene labeling. IEEE Trans Pattern Anal Mach Intell 35:1915–1929. https://doi.org/10.1109/TPAMI.2012.231

    Article  Google Scholar 

  62. 62.

    Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: IEEE transactions on pattern analysis and machine intelligence, pp 3431–3440

  63. 63.

    Zhang S, Fu H, Yan Y et al (2019) Attention guided network for retinal image segmentation. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 11764:797–805. https://doi.org/10.1007/978-3-030-32239-7_88

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Vedhus Hoskere.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hoskere, V., Narazaki, Y., Hoang, T.A. et al. MaDnet: multi-task semantic segmentation of multiple types of structural materials and damage in images of civil infrastructure. J Civil Struct Health Monit 10, 757–773 (2020). https://doi.org/10.1007/s13349-020-00409-0

Download citation

Keywords

  • Damage detection
  • Computer vision
  • Multi-task learning
  • Semantic segmentation
  • Structural inspections