Abstract
Manual visual inspection is the most common means of assessing the condition of civil infrastructure in the United States, but can be exceedingly laborious, time-consuming, and dangerous. Research has focused on automating parts of the inspection process using unmanned aerial vehicles for image acquisition, followed by deep learning techniques for damage identification. Existing deep learning methods and datasets for inspections have typically been developed for a single damage type. However, most guidelines for inspections require the identification of multiple damage types and describe evaluating the significance of the damage based on the associated material type. Thus, the identification of material type is important in understanding the meaning of the identified damage. Training separate networks for the tasks of material and damage identification fails to incorporate this intrinsic interdependence between them. We hypothesize that a network that incorporates such interdependence directly will have a better accuracy in material and damage identification. To this end, a deep neural network, termed the material-and-damage-network (MaDnet), is proposed to simultaneously identify material type (concrete, steel, asphalt), as well as fine (cracks, exposed rebar) and coarse (spalling, corrosion) structural damage. In this approach, semantic segmentation (i.e., assignment of each pixel in the image with a material and damage label) is employed, where the interdependence between material and damage is incorporated through shared filters learned through multi-objective optimization. A new dataset with pixel-level labels identifying the material and damage type is developed and made available to the research community. Finally, the dataset is used to evaluate MaDnet and demonstrate the improvement in pixel accuracy over employing independent networks.
References
Koch C, Georgieva K, Kasireddy V et al (2015) A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure. Adv Eng Inf 29:196–210
National Transportation Safety Board (NTSB) (2007) Collapse of I-35W highway bridge Minneapolis, Minnesota August 1, 2007. Highw Accid Rep 178:86
ATC (Applied Technology Council) (2005) Field manual: postearthquake safety evaluation of building. ATC-20-1, Redwood City, CA
Marroquin A (2017) Inspections bring high degree of difficulty at Hoover Dam bypass bridge. In: Las Vegas Rev. https://www.reviewjournal.com/local/local-nevada/inspections-bring-high-degree-of-difficulty-at-hoover-dam-bypass-bridge/. Accessed 23 Aug 2017
Abdel-Qader I, Abudayyeh O, Kelly ME (2003) Analysis of edge-detection techniques for crack identification in bridges. J Comput Civ Eng 17:255–263. https://doi.org/10.1061/(ASCE)0887-3801(2003)17:4(255)
Jahanshahi MR, Kelly JS, Masri SF, Sukhatme GS (2009) A survey and evaluation of promising approaches for automatic image-based defect detection of bridge structures. Struct Infrastruct Eng 5:455–486. https://doi.org/10.1080/15732470801945930
Yamaguchi T, Hashimoto S (2010) Fast crack detection method for large-size concrete surface images using percolation-based image processing. Mach Vis Appl 21:797–809. https://doi.org/10.1007/s00138-009-0189-8
Nishikawa T, Yoshida J, Sugiyama T, Fujino Y (2012) Concrete crack detection by multiple sequential image filtering. Comput Civ Infrastruct Eng 27:29–47. https://doi.org/10.1111/j.1467-8667.2011.00716.x
Zhu Z, German S, Brilakis I (2011) Visual retrieval of concrete crack properties for automated post-earthquake structural safety evaluation. Autom Constr 20:874–883. https://doi.org/10.1016/j.autcon.2011.03.004
Paal SG, Jeon J-S, Brilakis I, DesRoches R (2015) Automated damage index estimation of reinforced concrete columns for post-earthquake evaluations. J Struct Eng 141:04014228. https://doi.org/10.1061/(ASCE)ST.1943-541X.0001200
Adhikari RS, Moselhi O, Bagchi A (2013) A study of image-based element condition index for bridge inspection. In: ISARC 2013—30th Int Symp Autom Robot Constr Mining, Held Conjunction with 23rd World Min Congr, pp 345–356
Chen PH, Shen HK, Lei CY, Chang LM (2012) Support-vector-machine-based method for automated steel bridge rust assessment. Autom Constr 23:9–19. https://doi.org/10.1016/j.autcon.2011.12.001
Bonnin-Pascual F, Ortiz A (2014) Corrosion detection for automated visual inspection. Dev Corros Prot. https://doi.org/10.5772/57209
Yeum CM, Dyke SJ (2015) Vision-based automated crack detection for bridge inspection. Comput Civ Infrastruct Eng 30:759–770. https://doi.org/10.1111/mice.12141
Hu Y, Zhao C (2010) A novel LBP based methods for pavement crack detection. J Pattern Recognit Res 5:140–147. https://doi.org/10.13176/11.167
Zhang W, Zhang Z, Qi D, Liu Y (2014) Automatic crack detection and classification method for subway tunnel safety monitoring. Sensors (Switzerl) 14:19307–19328. https://doi.org/10.3390/s141019307
Wu S, Zhong S, Liu Y (2017) Deep residual learning for image steganalysis. Multimed Tools Appl 2017:1–17
Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. Int Conf Learn Represent. https://doi.org/10.1016/j.infsof.2008.09.005
LeCun Y, Bengio Y, Hinton G et al (2015) Deep learning. Nature 521:436–444. https://doi.org/10.1038/nature14539
Banga S, Gehani H, Bhilare S, et al (2018) 3D Topology Optimization using Convolutional Neural Networks. arXiv preprint arXiv:1808.07440v1
Lei X, Liu C, Du Z et al (2019) Machine learning-driven real-time topology optimization under moving morphable component-based framework. J Appl Mech Trans ASME. https://doi.org/10.1115/1.4041319
Sosnovik I, Oseledets I (2019) Neural networks for topology optimization. Russ J Numer Anal Math Model 34:215–223. https://doi.org/10.1515/rnam-2019-0018
Hoskere V, Eick BA, Spencer BF, Smith MD, Foltz SD (2019) Deep Bayesian neural networks for damage quantification in miter gates of navigation locks. Struct Heal Monit. https://doi.org/10.1177/1475921719882086
Rafiei MH, Adeli H (2018) A novel unsupervised deep learning model for global and local health condition assessment of structures. Eng Struct 156:598–607. https://doi.org/10.1016/j.engstruct.2017.10.070
Bao Y, Tang Z, Li H, Zhang Y (2019) Computer vision and deep learning–based data anomaly detection method for structural health monitoring. Struct Heal Monit 18:401–421. https://doi.org/10.1177/1475921718757405
Ye XW, Jin T, Yun CB (2019) A review on deep learning—based structural health monitoring of civil infrastructures. Smart Struct Syst 24:567–586. https://doi.org/10.12989/sss.2019.24.5.567
Tsuchimoto K, Narazaki Y, Hoskere V, Spencer Jr. BF (2020) Rapid postearthquake safety evaluation of buildings using sparse acceleration measurements. Struct Heal Monit (in press)
Dorafshan S, Maguire M (2018) Bridge inspection: human performance, unmanned aerial systems and automation. J Civ Struct Heal Monit 8:443–476. https://doi.org/10.1007/s13349-018-0285-4
Chen F-C, Jahanshahi MR, Johnson D, Delp EJ (2019) Vision-based decision support for flood risk assessment using google street view images. Struct Heal Monit. https://doi.org/10.12783/SHM2019/32472
Alipour M, Harris DK (2020) A big data analytics strategy for scalable urban infrastructure condition assessment using semi-supervised multi-transform self-training. J Civ Struct Heal Monit 10:313–332. https://doi.org/10.1007/s13349-020-00386-4
Yeum CM, Dyke SJ, Ramirez J (2018) Visual data classification in post-event building reconnaissance. Eng Struct 155:16–24. https://doi.org/10.1016/J.ENGSTRUCT.2017.10.057
Dimitrov A, Golparvar-Fard M (2014) Vision-based material recognition for automated monitoring of construction progress and generating building information modeling from unordered site image collections. Adv Eng Informatics 28:37–49. https://doi.org/10.1016/j.aei.2013.11.002
Roberts D, Torres Calderon W, Tang S, Golparvar-Fard M (2020) Vision-based construction worker activity analysis informed by body posture. J Comput Civ Eng 34:04020017. https://doi.org/10.1061/(asce)cp.1943-5487.0000898
Fang Q, Li H, Luo X et al (2018) A deep learning-based method for detecting non-certified work on construction sites. Adv Eng Informatics 35:56–68. https://doi.org/10.1016/j.aei.2018.01.001
Kim H, Kim H, Hong YW, Byun H (2018) Detecting construction equipment using a region-based fully convolutional network and transfer learning. J Comput Civ Eng 32:04017082. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000731
Yeum CM, Choi J, Dyke SJ (2017) Autonomous image localization for visual inspection of civil infrastructure. Smart Mater Struct 26:035051. https://doi.org/10.1088/1361-665X/aa510e
Narazaki Y, Hoskere V, Hoang TA, Spencer Jr. BF (2017) Vision-based automated bridge component recognition integrated with high-level scene understanding. In: The 13th international workshop on advanced smart materials and smart structures technology (ANCRiSST), Tokyo, Japan
Narazaki Y, Hoskere V, Hoang TA et al (2019) Vision-based automated bridge component recognition with high-level scene consistency. Comput Civ Infrastruct Eng mice 2019:12505. https://doi.org/10.1111/mice.12505
Narazaki Y, Hoskere V, Hoang TA, Spencer Jr. BF (2018) Automated bridge component recognition using video data. In: The 7th world conference on structural control and monitoring, 7WCSCM
Zhang L, Yang F, Daniel Zhang Y, Zhu YJ (2016) Road crack detection using deep convolutional neural network. IEEE Int Conf Image Process 2016:3708–3712. https://doi.org/10.1109/ICIP.2016.7533052
Yeum CM (2016) Computer vision-based structural assessment exploiting large volumes of images. Theses Diss Available from ProQuest
Hoskere V, Narazaki Y, Hoang TA, Spencer BF (2017) Vision-based Structural Inspection using Multiscale Deep Convolutional Neural Networks. In: 3rd Huixian International Forum on Earthquake Engineering for Young Researchers, University of Illinois, Urbana-Champaign. Urbana, Il, USA
Cha YJ, Choi W, Suh G et al (2018) Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Comput Civ Infrastruct Eng 33:731–747. https://doi.org/10.1111/mice.12334
Rubio JJ, Kashiwa T, Laiteerapong T et al (2019) Multi-class structural damage segmentation using fully convolutional networks. Comput Ind 112:103121. https://doi.org/10.1016/j.compind.2019.08.002
Spencer BF, Hoskere V, Narazaki Y (2019) Advances in computer vision-based civil infrastructure inspection and monitoring. Engineering 5:199–222
FHWA (2004) National bridge inspection standards regulations (NBIS). Fed Regist 69:15–35
Caruana R (1997) Multitask learning. Mach Learn 28:41–75. https://doi.org/10.1023/A:1007379606734
Kendall A, Gal Y, Cipolla R (2017) Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. Mach Learn. https://doi.org/10.1109/CVPR.2018.00781
Teichmann M, Weber M, Zöllner M et al (2018) MultiNet: real-time joint semantic reasoning for autonomous driving. In: IEEE intelligent vehicles symposium, proceedings. institute of electrical and electronics engineers Inc., pp 1013–1020
Moeskops P, Wolterink JM, van der Velden BHM et al (2016) Deep learning for multi-task medical image segmentation in multiple modalities. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics). Springer, pp 478–486
Facial Landmark Detection by Deep Multi-task Learning (2019) https://mmlab.ie.cuhk.edu.hk/projects/TCDCN.html. Accessed 15 Nov 2019
Shelhamer E, Long J, Darrell T et al (2015) Fully convolutional networks for semantic segmentation. IEEE Trans Pattern Anal Mach Intell 2015:640–651
Krogh A, Hertz JA (1992) A simple weight decay can improve generalization. Adv Neural Inf Process Syst 4:950–957
Kingma DP, Ba JL (2015) Adam: a method for stochastic optimization. Int Conf Learn Represent 2015:1–15. https://doi.org/10.1145/1830483.1830503
Abadi M, Barham P, Chen J et al (2016) TensorFlow: a system for large-scale machine learning. In: 12th symposium on operating systems design and implementation (2016). GA, pp 265–283
Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 580–587
Hoskere V, Narazaki Y, Hoang TA, Spencer Jr. BF (2017) Vision-based structural inspection using multiscale deep convolutional neural networks. In: 3rd Huixian International Forum on Earthquake Engineering for Young Researchers, University of Illinois, Urbana-Champaign
Chungwook S, Enrique V, Jhon PS et al (2016) Performance of low-rise reinforced concrete buildings in the 2016 Ecuador earthquake. https://datacenterhub.org/resources/14160. Accessed 6 Aug 2017
Bridgehunter.com: Historic Bridges of the United States (2017) https://bridgehunter.com/. Accessed 6 Aug 2017
US Army Corps of Engineers (2017) https://www.usace.army.mil/. Accessed 6 Aug 2017
Farabet C, Couprie C, Najman L, Lecun Y (2013) Learning hierarchical features for scene labeling. IEEE Trans Pattern Anal Mach Intell 35:1915–1929. https://doi.org/10.1109/TPAMI.2012.231
Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: IEEE transactions on pattern analysis and machine intelligence, pp 3431–3440
Zhang S, Fu H, Yan Y et al (2019) Attention guided network for retinal image segmentation. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 11764:797–805. https://doi.org/10.1007/978-3-030-32239-7_88
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Hoskere, V., Narazaki, Y., Hoang, T.A. et al. MaDnet: multi-task semantic segmentation of multiple types of structural materials and damage in images of civil infrastructure. J Civil Struct Health Monit 10, 757–773 (2020). https://doi.org/10.1007/s13349-020-00409-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13349-020-00409-0