Skip to main content
SpringerLink
Log in

Vision-based multi-level synthetical evaluation of seismic damage for RC structural components: a multi-task learning approach

  • Published:
Earthquake Engineering and Engineering Vibration Aims and scope Submit manuscript

Abstract

Recent studies for computer vision and deep learning-based, post-earthquake inspections on RC structures mainly perform well for specific tasks, while the trained models must be fine-tuned and re-trained when facing new tasks and datasets, which is inevitably time-consuming. This study proposes a multi-task learning approach that simultaneously accomplishes the semantic segmentation of seven-type structural components, three-type seismic damage, and four-type deterioration states. The proposed method contains a CNN-based encoder-decoder backbone subnetwork with skip-connection modules and a multi-head, task-specific recognition subnetwork. The backbone subnetwork is designed to extract multi-level features of post-earthquake RC structures. The multi-head, task-specific recognition subnetwork consists of three individual self-attention pipelines, each of which utilizes extracted multi-level features from the backbone network as a mutual guidance for the individual segmentation task. A synthetical loss function is designed with real-time adaptive coefficients to balance multi-task losses and focus on the most unstably fluctuating one. Ablation experiments and comparative studies are further conducted to demonstrate their effectiveness and necessity. The results show that the proposed method can simultaneously recognize different structural components, seismic damage, and deterioration states, and that the overall performance of the three-task learning models gains general improvement when compared to all single-task and dual-task models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Alipour M and Harris DK (2020), “Increasing the Robustness of Material-Specific Deep Learning Models for Crack Detection Across Different Materials,” Engineering Structures, 206: 110157.

    Article  Google Scholar 

  • Bao Y, Chen Z, Wei S, Tang Z, Xu Y and Li H (2019), “The State of The Art of Data Science and Engineering in Structural Health Monitoring,” Engineering, 5(2): 234–242.

    Article  Google Scholar 

  • Bao Y, Li J, Nagayama T, Xu Y, SpencerJr BF and Li H (2021), “The 1st International Project Competition for Structural Health Monitoring (IPC-SHM, 2020): A summary and Benchmark Problem,” Structural Health Monitoring, 20(4): 2229–2239.

    Article  Google Scholar 

  • Bello I, Zoph B, Vaswani A, Shlens J and Le QV (2019), “Attention Augmented Convolutional Networks,” Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3286–3295.

  • Cha YJ, Choi W and Büyüköztürk O (2017), “Deep Learning-based Crack Damage Detection using Convolutional Neural Networks,” Computer-Aided Civil and Infrastructure Engineering, 32(5): 361–378.

    Article  Google Scholar 

  • Chen FC and Jahanshahi MR (2017), “NB-CNN: Deep Learning-based Crack Detection using Convolutional Neural Network and Naïve Bayes Data Fusion,” IEEE Transactions on Industrial Electronics, 65(5): 4392–4400.

    Article  Google Scholar 

  • Chiozzi A and Miranda E (2017), “Fragility Functions for Masonry Infill Walls with In-plane Loading,” Earthquake Engineering & Structural Dynamics, 46(15): 2831–2850.

    Article  Google Scholar 

  • Choi W and Cha YJ (2019), “SDDNet: Real-Time Crack Segmentation,” IEEE Transactions on Industrial Electronics, 67(9): 8016–8025.

    Article  Google Scholar 

  • Cordonnier JB, Loukas A and Jaggi M (2019). “On the Relationship between Self-attention and Convolutional Layers,” arXiv preprint arXiv:1911.03584.

  • Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J and Houlsby N (2020), “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” arXiv preprint arXiv:2010.11929.

  • Duarte D, Nex F, Kerle N and Vosselman G (2018), “Satellite Image Classification of Building Damages using Airborne and Satellite Image Samples in a Deep Learning Approach,” ISPRS Annals of Photogrammetry, Remote Sensing & Spatial Information Sciences, 4(2).

  • Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z and Lu H (2019), “Dual Attention Network for Scene Segmentation,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3146–3154.

    Google Scholar 

  • Gao Y and Mosalam KM (2020), “PEER Hub ImageNet: A Large-Scale Multiattribute Benchmark Data Set of Structural Images,” Journal of Structural Engineering, 146(10), 04020198.

    Article  Google Scholar 

  • Guo MH, Xu TX, Liu JJ, Liu ZN, Jiang PT, Mu TJ, Zhang SH, Martin RR, Cheng MM and Hu SM (2022), “Attention Mechanisms in Computer Vision: A Survey,” Computational Visual Media, 1–38.

    Google Scholar 

  • Guo P, Lee CY and Ulbricht D (2020), “Learning to Branch for Multi-task Learning,” International Conference on Machine Learning, pp. 3854–3863.

  • Hoskere V (2020), “Developing Autonomy in Structural Inspections Through Computer Vision and Graphics,” PhD Dissertation, University of Illinois at Urbana-Champaign.

  • Hoskere V, Narazaki Y and SpencerJr BF (2022), “Physics-based Graphics Models in 3D Synthetic Environments as Autonomous Vision-based Inspection Testbeds,” Sensors, 22(2): 532.

    Article  Google Scholar 

  • Hoskere V, Narazaki Y and Spencer BF (2023), “Digital Twins as Testbeds for Vision-based Post-earthquake Inspections of Buildings,” European Workshop on Structural Health Monitoring, pp. 485–495, Springer, Cham.

    Google Scholar 

  • Hu H, Gu J, Zhang Z, Dai J and Wei Y (2018), “Relation Networks for Object Detection,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3588–3597.

  • Kakooei M and Baleghi Y (2017), “Fusion of Satellite, Aircraft, and UAV Data for Automatic Disaster Damage Assessment,” International Journal of Remote Sensing, 38(8–10): 2511–2534.

    Article  Google Scholar 

  • Kendall A, Gal Y and Cipolla R (2018), “Multi-task Learning using Uncertainty to Weigh Losses for Scene Geometry and Semantics,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7482–7491.

  • Kim H, Yoon J and Sim SH (2020), “Automated Bridge Component Recognition from Point Clouds Using Deep Learning,” Structural Control and Health Monitoring, 27(9): e2591.

    Article  Google Scholar 

  • Kingma DP and Ba J (2014), “Adam: A Method for Stochastic Optimization,” arXiv preprint arXv.1412.6980.

  • Li H, Spencer BF, Bao Y, et al. (2021), “Machine Learning Paradigm for Structural Health Monitoring,” Structural Health Monitoring, 20(4): 1353–1372.

    Article  Google Scholar 

  • Li S, Zhao X and Zhou G (2019), “Automatic Pixel-Level Multiple Damage Detection of Concrete Structure Using Fully Convolutional Network,” Computer-Aided Civil and Infrastructure Engineering, 34(7): 616–634.

    Article  Google Scholar 

  • Liu PCY and El-Gohary N (2020), “Semantic Image Retrieval and Clustering for Supporting Domain-Specific Bridge Component and Defect Classification,” Construction Research Congress 2020: Infrastructure Systems and Sustainability, pp. 809–818. Reston, VA: American Society of Civil Engineers.

    Google Scholar 

  • Liu S, Johns E and Davison AJ (2019), “End-to-end Multi-task Learning with Attention,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1871–1880.

  • Misra I, Shrivastava A, Gupta A and Hebert M (2016), “Cross-Stitch Networks for Multi-task Learning,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994–4003.

  • Narazaki Y, Hoskere V, Hoang TA, Fujino Y, Sakurai A and SpencerJr BF (2020), “Vision-based Automated Bridge Component Recognition with High-level Scene Consistency,” Computer-Aided Civil and Infrastructure Engineering, 35(5): 465–482.

    Article  Google Scholar 

  • Narazaki Y, Hoskere V, Yoshida K, SpencerJr BF and Fujino Y (2021), “Synthetic Environments for Vision-based Structural Condition Assessment of Japanese High-speed Railway Viaducts,” Mechanical Systems and Signal Processing, 160: 107850.

    Article  Google Scholar 

  • Pan Y, Zhang G and Zhang L (2020), “A Spatial-channel Hierarchical Deep Learning Network for Pixel-level Automated Crack Detection,” Automation in Construction, 119: 103357.

    Article  Google Scholar 

  • Protopapadakis E, Makantasis K, Kopsiaftis G, Doulamis N and Amditis A (2016), “Crack Identification via User Feedback, Convolutional Neural Networks and Laser Scanners for Tunnel Infrastructures,” International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 5: 725–734. SCITEPRESS.

    Article  Google Scholar 

  • Rich C (1997), “Multi-task Learning,” Machine Learning, 28: 41–75.

    Article  Google Scholar 

  • Ros G, Sellart L, Materzynska J, Vazquez D and Lopez AM (2016), “The Synthia Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3234–3243.

  • Shao J, Tang L, Liu M, Shao G, Sun L and Qiu Q (2020), “BDD-Net: A General Protocol for Mapping Buildings Damaged by a Wide Range of Disasters Based on Satellite Imagery,” Remote Sensing, 12(10): 1670.

    Article  Google Scholar 

  • Son H and Kim C (2010), “3D Structural Component Recognition and Modeling Method using Color and 3D Data for Construction Progress Monitoring,” Automation in Construction, 19(7): 844–854.

    Article  Google Scholar 

  • Soukup D and Huber-Mörk R (2014), “Convolutional Neural Networks for Steel Surface Defect Detection from Photometric Stereo Images,” International Symposium on Visual Computing, pp. 668–677. Springer, Cham.

    Book  Google Scholar 

  • SpencerJr BF, Hoskere V and Narazaki Y (2019), “Advances in Computer Vision-based Civil Infrastructure Inspection and Monitoring,” Engineering, 5(2): 199–222.

    Article  Google Scholar 

  • Wang Y, Cui L, Zhang C, Chen W, Xu Y and Zhang Q (2022), “A Two-stage Seismic Damage Assessment Method for Small, Dense, and Imbalanced Buildings in Remote Sensing Images,” Remote Sensing, 14(4): 1012.

    Article  Google Scholar 

  • Xiong C, Li Q and Lu X (2020), “Automated Regional Seismic Damage Assessment of Buildings using An Unmanned Aerial Vehicle and a Convolutional Neural Network,” Automation in Construction, 109: 102994.

    Article  Google Scholar 

  • Xu Y, Bao Y, Chen J, Zuo W and Li H (2019a), “Surface Fatigue Crack Identification in Steel Box Girder of Bridges by A Deep Fusion Convolutional Neural Network Based on Consumer-grade Camera Images,” Structural Health Monitoring, 18(3): 653–674.

    Article  Google Scholar 

  • Xu Y, Bao Y, Zhang Y and Li H (2020), “Attribute-Based Structural Damage Identification by Few-shot Meta Learning with Inter-class Knowledge Transfer,” Structural Health Monitoring, 20(4): 1494–1517.

    Article  Google Scholar 

  • Xu Y, Li S, Zhang D, Jin Y, Zhang F, Li N and Li H (2018), “Identification Framework for Cracks on A Steel Structure Surface by A Restricted Boltzmann Machines Algorithm Based on Consumer-Grade Camera Images,” Structural Control and Health Monitoring, 25(2): e2075.

    Article  Google Scholar 

  • Xu Y, Wei S, Bao Y and Li H (2019b), “Automatic Seismic Damage Identification of Reinforced Concrete Columns from Images by a Region-Based Deep Convolutional Neural Network,” Structural Control and Health Monitoring, 26(3): e2313.

    Article  Google Scholar 

  • Zhang A, Wang KC, Li B, Yang E, Dai X, Peng Y, Fei Y, Liu Y, Li J and Chen C (2017), “Automated Pixel-level Pavement Crack Detection on 3D Asphalt Surfaces Using a Deep-learning Network,” Computer-Aided Civil and Infrastructure Engineering, 32(10): 805–819.

    Article  Google Scholar 

  • Zhang Y and Yang Q (2017), “A Survey on Multi-task Learning,” arXiv preprint arXiv:1707.08114.

  • Zhao J, Hu F, Qiao W, Zhai W, Xu Y, Bao Y and Li H (2022), “A Modified U-Net for Crack Segmentation by Self-Attention-Self-Adaption Neuron and Random Elastic Deformation,” Smart Structures and Systems, 29(1): 1–16.

    Google Scholar 

  • Zhao J, Hu F, Xu Y, Zuo W, Zhong J and Li H (2022), “Structure-PoseNet for Identification of Dense Dynamic Displacement and Three-dimensional Poses of Structures Using a Monocular Camera,” Computer-Aided Civil and Infrastructure Engineering, 37(6): 704–725.

    Article  Google Scholar 

  • Zhang LX, Shen JK and Zhu BJ (2022), “A Review of the Research and Application of Deep Learning-Based Computer Vision in Structural Damage Detection,” Earthquake Engineering and Engineering Vibration, 21(1): 1–21.

    Article  Google Scholar 

Download references

Acknowledgement

Financial support for this study was provided by the National Key R&D Program of China (Grant No. 2019YFC1511005), the National Natural Science Foundation of China (Grant Nos. 51921006, 52192661, and 52008138), the China Postdoctoral Science Foundation (Grant Nos. BX20190102 and 2019M661286), the Heilongjiang Natural Science Foundation (Grant No. LH2022E070), and the Heilongjiang Province Postdoctoral Science Foundation (Grant Nos. LBH-TZ2016 and LBH-Z19064).

Author information

Authors and Affiliations

  1. Key Lab of Smart Prevention and Mitigation of Civil Engineering Disasters of the Ministry of Industry and Information Technology, Harbin Institute of Technology, Harbin, 150090, China

    Yang Xu, Weidong Qiao, Jin Zhao & Hui Li

  2. Key Lab of Structures Dynamics Behavior and Control of the Ministry of Education, Harbin Institute of Technology, Harbin, 150090, China

    Yang Xu, Weidong Qiao, Jin Zhao & Hui Li

  3. School of Civil Engineering, Harbin Institute of Technology, Harbin, 150090, China

    Yang Xu, Weidong Qiao, Jin Zhao & Hui Li

  4. School of Civil Engineering and Mechanics, Lanzhou University, Lanzhou, 730000, China

    Qiangqiang Zhang

Authors
  1. Yang Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weidong Qiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jin Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qiangqiang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yang Xu.

Additional information

Supported by: National Key R&D Program of China under Grant No. 2019YFC1511005, the National Natural Science Foundation of China under Grant Nos. 51921006, 52192661 and 52008138, the China Postdoctoral Science Foundation under Grant Nos. BX20190102 and 2019M661286, the Heilongjiang Natural Science Foundation under Grant No. LH2022E070, and the Heilongjiang Province Postdoctoral Science Foundation under Grant Nos. LBH-TZ2016 and LBH-Z19064

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, Y., Qiao, W., Zhao, J. et al. Vision-based multi-level synthetical evaluation of seismic damage for RC structural components: a multi-task learning approach. Earthq. Eng. Eng. Vib. 22, 69–85 (2023). https://doi.org/10.1007/s11803-023-2153-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11803-023-2153-4

Keywords

Search

Navigation