Skip to main content
SpringerLink
Log in

A deep reinforcement learning framework for life-cycle maintenance planning of regional deteriorating bridges using inspection data

  • Research Paper
  • Published:
Structural and Multidisciplinary Optimization Aims and scope Submit manuscript

Abstract

Determination of regional deteriorating bridges’ maintenance strategies for minimizing life-cycle risks and costs constructs a complex optimization problem. Improper maintenance strategies lead to budget waste and ineffective maintenances. To optimize the life-cycle maintenance strategies of regional bridges, this study develops a deep reinforcement learning (DRL)-based framework to enable the agent to learn better maintenance actions in an interactive environment by trial and error. It is able to generate several optimal maintenance strategies to match with different budget constraints under the premise of increasing the maintenance cost-effectiveness to the greatest extent possible. The framework contains the whole optimization process from data collection to regional functions establishment to reinforcement learning training. The regional structural deterioration features and the effect of maintenance actions are determined by years of regional inspection reports. The regional probabilistic models are incorporated to simulate the stochastic process. The regional life-cycle maintenance strategies are optimized with the deep Q-networks model. The proposed framework is substantiated by developing 100-year maintenance strategies for part of highway bridges. The results show that the trained optimal maintenance strategies match well with the given budget constraints and maximize the life-cycle cost-effectiveness of maintenance actions. By employing the trained maintenance strategies, the conditions of regional bridges are controlled at a good level. Besides, the trained optimal DRL-based strategies have better performance than traditional condition-fixed strategies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

References

  • Akgul F (2013) Development of a bridge management system incorporating a newly developed model for element condition evaluation based on damage effects. Struct Infrastruct Eng 9:1206–1224

    Google Scholar 

  • Andriotis CP, Papakonstantinou KG (2021) Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints. Reliab Eng Syst Saf 212:107551

    Google Scholar 

  • Anwar GA, Dong Y, Zhai CH (2020) Performance-based probabilistic framework for seismic risk, resilience, and sustainability assessment of reinforced concrete structures. Adv Struct Eng 23:1454–1472

    Google Scholar 

  • Arulkumaran K, Deisenroth MP, Brundage M, Bharath AA (2017) Deep reinforcement learning a brief survey. IEEE Signal Process Mag 34:26–38

    Google Scholar 

  • Bu GP, Lee JH, Guan H, Loo YC, Blumenstein M (2015) Prediction of long-term bridge performance: integrated deterioration approach with case studies. J Perform Constr Facil 29:04014089

    Google Scholar 

  • Bukhsh ZA, Stipanovic I, Klanker G, Oonnor A, Doree AG (2019) Network level bridges maintenance planning using multi-attribute utility theory. Struct Infrastruct Eng 15:872–885

    Google Scholar 

  • Bukhsh ZA, Stipanovic I, Saeed A, Doree AG (2020) Maintenance intervention predictions using entity-embedding neural networks. Autom Constr 116:103202

    Google Scholar 

  • Busoniu L, Babuska R, De Schutter B (2008) A comprehensive survey of multiagent reinforcement learning. IEEE Trans Syst Man Cybernet Part C—Appl Rev 38:156–172

    Google Scholar 

  • Calvert G, Neves L, Andrews J, Hamer M (2020) Multi-defect modelling of bridge deterioration using truncated inspection records. Reliab Eng Syst Saf 200:106962

    Google Scholar 

  • Chen JQ, Zhang XS, Jing Z (2018) A cooperative PSO-DP approach for the maintenance planning and RBDO of deteriorating structures. Struct Multidisc Optim 58:95–113

    MathSciNet  Google Scholar 

  • Cheng MY, Chiu YF, Chiu CK, Prayogo D, Wu YW, Hsu ZL, Lin CH (2019) Risk-based maintenance strategy for deteriorating bridges using a hybrid computational intelligence technique: a case study. Struct Infrastruct Eng 15:334–350

    Google Scholar 

  • Chhabra JPS, Warn GP (2019) A method for model selection using reinforcement learning when viewing design as a sequential decision process. Struct Multidisc Optim 59:1521–1542

    MathSciNet  Google Scholar 

  • Contreras-Nieto C, Shan YW, Lewis P, Hartell JA (2019) Bridge maintenance prioritization using analytic hierarchy process and fusion tables. Autom Constr 101:99–110

    Google Scholar 

  • Dong Y, Frangopol DM, Saydam D (2013) Time-variant sustainability assessment of seismically vulnerable bridges subjected to multiple hazards. Earthquake Eng Struct Dynam 42:1451–1467

    Google Scholar 

  • Fiorillo G, Nassif H (2020) Improving the conversion accuracy between bridge element conditions and NBI ratings using deep convolutional neural networks. Struct Infrastruct Eng 16:1669–1682

    Google Scholar 

  • Frangopol DM, Liu M (2007) Maintenance and management of civil infrastructure based on condition, safety, optimization, and life-cycle cost. Struct Infrastruct Eng 3:29–41

    Google Scholar 

  • Frangopol DM, Dong Y, Sabatino S (2017) Bridge life-cycle performance and cost: analysis, prediction, optimisation and decision-making. Struct Infrastruct Eng 13:1239–1257

    Google Scholar 

  • Garcia-Segura T, Yepes V, Frangopol DM (2017) Multi-objective design of post-tensioned concrete road bridges using artificial neural networks. Struct Multidisc Optim 56:139–150

    MathSciNet  Google Scholar 

  • Gong CQ, Frangopol DM (2020) Condition-based multiobjective maintenance decision making for highway bridges considering risk perceptions. J Struct Eng 146:04020051

    Google Scholar 

  • Jeong Y, Kim W, Lee I, Lee J (2018) Bridge inspection practices and bridge management programs in China, Japan, Korea, and U.S. J Struct Integr Maint 3:126–135

    Google Scholar 

  • Kabir G, Sadiq R, Tesfamariam S (2014) A review of multi-criteria decision-making methods for infrastructure management. Struct Infrastruct Eng 10:1176–1210

    Google Scholar 

  • Khalatbarisoltani A, Soleymani M, Khodadadi M (2019) Online control of an active seismic system via reinforcement learning. Struct Control Health Monit 26:e2298

    Google Scholar 

  • Kim S, Frangopol DM (2018) Multi-objective probabilistic optimum monitoring planning considering fatigue damage detection, maintenance, reliability, service life and cost. Struct Multidisc Optim 57:39–54

    MathSciNet  Google Scholar 

  • Kim S, Ge BX, Frangopol DM (2019) Effective optimum maintenance planning with updating based on inspection information for fatigue-sensitive structures. Probab Eng Mech 58:103003

    Google Scholar 

  • Lei X, Sun L, Xia Y (2020) Lost data reconstruction for structural health monitoring using deep convolutional generative adversarial networks. Struct Health Monit 20(4):2069–2087

    Google Scholar 

  • Lei XM, Sun LM, Xia Y (2021) Seismic fragility assessment and maintenance management on regional bridges using bayesian multi-parameter estimation. Bull Earthq Eng 19:6693

    Google Scholar 

  • Li Z, Burgueno R (2010) Using soft computing to analyze inspection results for bridge evaluation and management. J Bridg Eng 15:430–438

    Google Scholar 

  • Liu H, Zhang YF (2020) Bridge condition rating data modeling using deep learning algorithm. Struct Infrastruct Eng 16:1447–1460

    Google Scholar 

  • Matos JC, Cruz PJS, Valente IB, Neves LC, Moreira VN (2016) An innovative framework for probabilistic-based structural assessment with an application to existing reinforced concrete structures. Eng Struct 111:552–564

    Google Scholar 

  • Matos JC, Moreira VN, Valente IB, Cruz PJS, Neves LC, Galvao N (2019) Probabilistic-based assessment of existing steel-concrete composite bridges application to Sousa River Bridge. Eng Struct 181:95–110

    Google Scholar 

  • Memarzadeh M, Pozzi M (2019) Model-free reinforcement learning with model-based safe exploration: optimizing adaptive recovery process of infrastructure systems. Struct Saf 80:46–55

    Google Scholar 

  • Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, Petersen S, Beattie C, Sadik A, Antonoglou I, King H, Kumaran D, Wierstra D, Legg S, Hassabis D (2015) Human-level control through deep reinforcement learning. Nature 518:529–533

    Google Scholar 

  • Orcesi AD, Frangopol DM (2011) Probability-based multiple-criteria optimization of bridge maintenance using monitoring and expected error in the decision process. Struct Multidisc Optim 44:137–148

    Google Scholar 

  • Ruiz E, Freyne S (2020) Multiple regression model for load rating of reinforced concrete bridges. Transp Res Rec 2674:361–372

    Google Scholar 

  • Sabatino S, Frangopol DM, Dong Y (2016) Life cycle utility-informed maintenance planning based on lifetime functions: optimum balancing of cost, failure consequences and performance benefit. Struct Infrastruct Eng 12:830–847

    Google Scholar 

  • Selvin S, Vinayakumar R, Gopalakrishnan EA, Menon VK, Soman KP (2017) Stock price prediction using LSTM, RNN and CNN-sliding window model. 2017 International conference on advances in computing, communications and informatics (Icacci). IEEE, Udupi, pp 1643–1647

    Google Scholar 

  • Sutton RS, Barto AG (2018) Reinforcement learning: an introduction, 2nd edn. The MIT Press, Cambridge

    MATH  Google Scholar 

  • Tao WF, Lin PH, Wang NY (2021) Optimum life-cycle maintenance strategies of deteriorating highway bridges subject to seismic hazard by a hybrid Markov decision process model. Struct Saf 89:102042

    Google Scholar 

  • van Noortwijk JM, Frangopol DM (2004) Two probabilistic life-cycle maintenance models for deteriorating civil infrastructures. Probab Eng Mech 19:345–359

    Google Scholar 

  • Wang Z, Cha Y-J (2020) Unsupervised deep learning approach using a deep auto-encoder with an one-class support vector machine to detect structural damage. Struct Health Monit 20(1):406–425

    Google Scholar 

  • Wei S, Bao Y, Li H (2020) Optimal policy for structure maintenance: a deep reinforcement learning framework. Struct Saf 83:101906

    Google Scholar 

  • Wu Y, Hou G, Chen S (2021) Post-earthquake resilience assessment and long-term restoration prioritization of transportation network. Reliab Eng Syst Saf 211:107612

    Google Scholar 

  • Xia Y, Lei XM, Wang P, Liu GM, Sun LM (2020) Long-term performance monitoring and assessment of concrete beam bridges using neutral axis indicator. Struct Control Health Monit 27:e2637

    Google Scholar 

  • Xia Y, Lei XM, Wang P, Sun LM (2021) Artificial intelligence based structural assessment for regional short- and medium-span concrete beam bridges with inspection information. Remote Sens 13:3687

    Google Scholar 

  • Xia Y, Lei X, Wang P, Sun L (2021) A data-driven approach for regional bridge condition assessment using inspection reports. Struct Control Health Monit. https://doi.org/10.1002/stc.2915

    Article  Google Scholar 

  • Xu X, Huang Q, Ren Y, Sun HB (2018) Condition assessment of suspension bridges using local variable weight and normal cloud model. KSCE J Civ Eng 22:4064–4072

    Google Scholar 

  • Xu X, Huang Q, Ren Y, Zhao DY, Zhang DY, Sun HB (2019) Condition evaluation of suspension bridges for maintenance, repair and rehabilitation: a comprehensive framework. Struct Infrastruct Eng 15:555–567

    Google Scholar 

  • Yang DY, Frangopol DM (2020) Life-cycle management of deteriorating bridge networks with network-level risk bounds and system reliability analysis. Struct Saf 83:101911

    Google Scholar 

  • Yao LY, Dong Q, Jiang JW, Ni FJ (2020) Deep reinforcement learning for long-term pavement maintenance planning. Comput Aided Civil Infrastruct Eng 35:1230–1245

    Google Scholar 

  • Yonekura K, Hattori H (2019) Framework for design optimization using deep reinforcement learning. Struct Multidisc Optim 60:1709–1713

    Google Scholar 

  • You CX, Lu JB, Filev D, Tsiotras P (2019) Advanced planning for autonomous vehicles using reinforcement learning and deep inverse reinforcement learning. Robot Auton Syst 114:1–18

    Google Scholar 

  • Yu Y, Hur T, Jung J, Jang IG (2019) Deep learning for determining a near-optimal topological design without any iteration. Struct Multidisc Optim 59:787–799

    Google Scholar 

  • Zhang NL, Si WJ (2020) Deep reinforcement learning for condition-based maintenance planning of multi-component systems under dependent competing risks. Reliab Eng Syst Saf 203:107094

    Google Scholar 

Download references

Funding

This paper is supported by the National Key Research and Development Program of China (2019YFB1600702), Transportation Science and Technology Program of Shandong Province (2021B51), National Natural Science Foundation of China (51978508), and Technology Cooperation Project of Shanghai Qizhi Institute (SYXF0120020109).

Author information

Authors and Affiliations

  1. Department of Bridge Engineering, Tongji University, Shanghai, 200092, China

    Xiaoming Lei, Ye Xia & Limin Sun

  2. Key Laboratory of Damage Diagnosis for Engineering Structures of Hunan Province, Hunan University, Changsha, 410082, China

    Lu Deng

  3. State Key Laboratory of Disaster Reduction in Civil Engineering, Tongji University, Shanghai, 200092, China

    Limin Sun

  4. Shanghai Qi-Zhi Research Institute, Shanghai, 200003, China

    Ye Xia & Limin Sun

  5. College of Civil Engineering, Hunan University, Changsha, 410082, China

    Lu Deng

Authors
  1. Xiaoming Lei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ye Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lu Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Limin Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ye Xia.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Replication of results

The Python codes that support the findings of this study are available upon reasonable request to the corresponding or first author.

Additional information

Responsible Editor: Chao Hu.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lei, X., Xia, Y., Deng, L. et al. A deep reinforcement learning framework for life-cycle maintenance planning of regional deteriorating bridges using inspection data. Struct Multidisc Optim 65, 149 (2022). https://doi.org/10.1007/s00158-022-03210-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00158-022-03210-3

Keywords

Search

Navigation